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This  report  documents  the  research  and  analysis  project  that  resulted  In 
the  development  of  Instructor  Support  Feature  (ISF)  Guidelines.  The 
guidelines  are  Intended  to  aid  operational  users  from  the  Mr  Force  major 
commands,  Simulator  Systems  Program  Office  procurement  personnel,  and 
contractors  In  the  development  and  procurement  of  Instructor  support  systems 
for  future  aircrew  training  devices  (ATDs).  During  the  12-month  technical 
effort,  the  Guidelines  content  and  format  were  defined,  data  were  collected 
and  analyzed  for  Inclusion  within  the  Guidelines,  and  the  Guidelines  document 
was  written.  Thirteen  advanced  instructional  systems  and  ATDs  provided  data 
for  Identification  and  definition  of  ISF  requirements.  Volume  I  documents  the 
research  and  development  effort  and  presents  methodology,  results,  conclu¬ 
sions,  and  recommendations.  Volume  II  contains  the  ISF  Guidelines.  The  ISF 
Guidelines  Is  a  "living”  document.  The  current  version  of  the  Guidelines  can 
be  obtained  from  the  Simulator  Systems  Program  Office,  ASD/YWEE, 
Wright-Patterson  AFB,  OH. 


PREFACE 


This  document  is  the  final  report  of  the  Performance  Measurement 
System  (PMS)  Guidelines  for  Aircrew  Training  Devices  (ATDs)  project 
conducted  under  Contract  Number  F33615-84-C-0054,  sponsored  by  the 
Air  Force  Human  Resources  Laboratory  (AFHRL).  The  project  focused  on 
the  development  of  the  Instructor  Support  Feature  (ISF)  Guidelines  to 
aid  in  the  specification  of  requirements  for  ATD  acquisitions.  The 
Guidelines  are  published  as  Volume  11  of  this  report. 

Drs.  Wayne  Waag  and  Gary  Thomas  of  AFHRL/OT  provided  technical 
direction  during  the  course  of  the  study.  Mr.  Craig  McLean  and  his 
staff  at  the  Simulator  Systems  Program  Office  made  valuable 
contributions  to  the  contents  of  the  1SF  Guidelines. 

The  authors  wish  to  express  their  gratitude  to  the  many 
operational  personnel  at  the  training  sites  visited  for  their  time 
and  assistance.  Their  input  greatly  added  to  the  operational 
validity  of  this  report. 
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I .  INTRODUCTION 


This  report  supplements  the  Instructor  Support  Feature  (ISF)  Guidelines. 
The  purpose  of  the  ISF  Guidelines  is  to  provide  a  communications  vehicle  among 
those  personnel  involved  in  the  acquisition  process  for  a  new  training  device. 
These  personnel  include  operational  users  at  the  Air  Force  Major  Commands 
(MAJCOMs)  who  initially  state  training  requirements,  Simulator  Systems  Program 
Office  (SimSPO)  procurement  personnel  involved  in  the  final  specification 
definition,  and  contracting  personnel  involved  in  system  development.  The 
Guidelines  specifically  provide  guidance  in  the  development  of  specifications 
for  the  "instructional  component"  within  the  total  simulation  system.  The 
research  and  development  activities  conducted  in  support  of  the  development  of 
the  Guidelines  revealed  several  interesting  and  notable  results  and  findings 
with  respect  to  the  use  of  aircrew  training  devices.  The  purpose  of  this 
report  is  to  document  these  conclusions  and  present  the  methods  used  to  reach 
them. 

\ 

*  .The  following  major  conclusions  were  reached: 

y  '  >  *  <<■*_•*  <*  I  '..-J  £ 

1.  A  comprehensive  front-end  analysis  of  both  student  training  tasks  and 
instructor  requirements  is  required*<4ji  order  to  ensure  that  the  instruc¬ 
tional  system  is  designed  to  meet  user  Qpeds. 

)  'J  < 

2;  Instructor  training  in  the  use  of  instructor  support  features  is 
needed. 


3.  The  iponcept  of  the  "task  module"  has  successfully  been  used  to  ensure 
that  operational  data  are  provided  so  that  the  resulting  instructional  system 
support»-«Ser  needs . 

( '  4.  Instructional  system  data  (*»£-<  maneuvers  and  procedures,  displays, 
and  performance  measurement  criteria)  must  be  kept  current  with  changes 
in  training  requirements  and  flight  procedures.!  -Provision  for  economic 
update  and  revision  is  crucial. 


[f.J  The  instructional  system  should  provide  for  level(s)  of  automated 
control  that  support  the  specific  training  objectives. 

'6.  The  Automated  Performance  Measurement  feature  should  be  designed  as 
an  aid,  not  as  a  replacement,  to  support  the  instructor  in  an  evaluation 
of  the  student's  performance  of  the  training  objective. 

7.  The  specification  of  instructional  features  should  be  based  on 
functionality  and  performance  from  a  user's  perspective  rather  than  on  the 
latest  technology.  Current  technological  advances  and  current  standards  can 
then  be  incorporated  to  properly  support  these  specified  functional  require¬ 
ments  . 


8.  These  Guidelines  should  be  continually  updated  so  that  lessons 
learned  and  proven  technology  from  advanced  instructional  systems  can  be 
effectively  transitioned  into  the  operational  training  environment. 

These  conclusions  are  addressed  in  greater  detail  in  Section  IV. 

Background 

In  1981,  the  SimSPO  of  the  Air  Force  Aeronautical  Systems  Division  (ASD) 
stated  a  need  for  enhancing  the  instructor's  capability  to  assess  student 
performance  in  ATDs .  The  need  for  improved  student  performance  measurement 
capabilities  within  ATDs  was  also  clearly  identified  by  the  Defense  Science 
Board  1982  Summer  Panel  Study  on  Training  and  Training  Technology. 

Prototype  systems,  incorporating  state-of-the-art  performance  measurement 
technology,  have  been  successfully  implemented  in  research  and  development 
environments  and  have  provided  valuable  lessons.  A  means  was  sought  for 
capitalizing  on  this  information  for  application  in  the  operational  training 
environment.  Development  of  a  set  of  guidelines  addressing  the  design, 
development,  and  incorporation  of  Performance  Measurement  System  (PMS)  capa¬ 
bilities  within  ATDs  was  the  proposed  solution.  By  providing  guidelines  for 
personnel  who  are  tasked  with  specifying  these  requirements  in  Prime  Item 
Development  Specifications,  proven  and  current  technology  could  be  effectively 
transitioned  into  the  operational  setting. 

The  scope  of  the  guidelines  was  defined  to  include  performance  measurement 
requirements  such  as  parameters  measurement,  associated  scores,  and  start-stop 
logic  and  also  instructor  support  requirements  such  as  scenario  control, 
briefing,  display  of  information,  and  debriefing.  Associated  computer  hardware 
and  software  considerations  were  included  as  well.  Instructor  support 
considerations  were  identified  early  on  in  the  project  as  essential  to  instruc¬ 
tor  acceptance  and  utilization  of  the  PMS  within  an  ATD.  Thus  it  became 
apparent  that  the  guidelines  must  feature  all  instructor  support  requirements, 
with  performance  measurement  as  one  of  these  requirements.  The  guidelines 
were  therefore  renamed  "Instructor  Support  Feature  (ISF)  Guidelines."  The  term 
"instructor  support  feature"  is  used  to  describe  those  capabilities  of  the  ATD 
that  are  specifically  designed  to  aid  instructional  activity.  The  term 
"instructor  support  system"  (ISS)  is  used  in  this  document  to  refer  to  computer- 
based  systems  which  support  instructor  support  features. 

Prototype  training  systems  have  demonstrated  that  an  ISS  can  provide  the 
instructor  with  greater  ability  to  control  and  monitor  student  activity  and 
therefore  to  make  the  simulator  a  more  effective  training  system.  These  systems 
have  much  to  offer  insofar  as  lessons  learned  during  their  development,  test 
and  evaluation,  and  operation.  The  lessons  learned  from  these  systems  as  well 
as  from  other  ATDs  can  contribute  dramatically  to  the  improvement  of  the 
specification  of  ATD  requirements. 


If  a  single  guidelines  document  could  be  used  by  the  spectrum  of 
individuals  involved  in  specifying  ATD  requirements,  it  would  serve  as  a 
common  basis  for  communication  of  need  and  would  promote  a  greater  degree  of 


mutual  understanding.  Current  ATD  acquisitions  utilize  specifications  for 
instructor  support  requirements  which  are  rudimentary  at  best.  This  has 
resulted  in  the  design  and  implementation  of  aircrew  training  devices  with 
features  which  are  either  too  difficult  for  instructors  to  use  or  do  not 
fulfill  the  training  needs.  These  deficiencies  result  in  low  utilization 
rates  by  instructors,  loss  of  productive  training  time,  and  ineffective 
training . 

In  summary,  the  major  goal  of  the  effort  was  the  development  of  a  set  of 
clear,  usable  guidelines  for  instructor  support  feature  requirements  in  future 
ATD  acquisitions.  These  guidelines  are  intended  to  be  used  by  MAJCOM  require¬ 
ments  personnel,  SimSPO  specification  writers,  ATD  users,  and  contractors 
alike.  It  is  anticipated  that  use  of  these  guidelines  will  result  in  well 
defined  specifications  that  lead  to  the  provision  of  useful  instructional 
support  capabilities  within  ATDs.  Secondary  goals  include  the  provision  of 
data  which  facilitate  the  further  definition  of  mission  tasks  and  instructional 
support  functions  and  the  promotion  of  more  effective  ISS  designs. 

The  remainder  of  the  report  is  divided  by  section.  Section  II,  Method, 
describes  how  the  study  activities  were  accomplished.  Section  III,  Results, 
documents  the  findings  of  the  investigations  and  analyses.  Section  IV, 
Conclusions  and  Recommendations,  discusses  overall  conclusions  derived  as  a 
result  of  these  activities  and  offers  recommendations  for  the  future. 


II.  METHOD 


This  section  describes  the  project  activities  that  were  conducted  to 
develop  the  ISF  Guidelines.  A  general  overview  is  followed  by  a  detailed 
description  of  the  major  activities.  The  results  of  these  efforts  are  described 
in  Section  III.  The  categories  in  Sections  II  (Methods)  and  III  (Results)  are 
presented  in  parallel  order. 


General  Overview 


The  project  goals  were  achieved  using  a  three-phased  study  approach: 

1.  Definition  of  design  guidelines  content  and  format. 

2.  Review  of  simulator  training  requirements  across  a  spectrum  of  Air 
Force  Commands,  and  review  of  the  state-of-the-art  capabilities  of  four 
systems  which  utilize  advanced  instructor  support  features. 

3.  Development  of  the  ISF  Guidelines  and  a  sample  specification. 

Figure  1  presents  the  overall  "roadmap"  of  the  approach.  The  three 
phases  of  the  study  were,  for  the  most  part,  time-phased,  with  some  overlap  in 
activities. 

The  project  began  during  Phase  I  with  the  identification  of  prospective 
content  and  format  for  the  guidelines.  In  order  to  accomplish  this,  the 
specification  process  and  relevant  sections  of  specifications  from  recent  ATD 
acquisitions  were  reviewed.  In  addition,  a  meeting  with  SimSPO  and  MAJCOM 
personnel  was  held  to  determine  needs.  A  library  of  materials,  including 
course  documents,  systems  documentation,  and  ISF-related  information,  was  also 
used  to  determine  appropriate  content  and  potential  formats. 

Phase  II  encompassed  a  review  of  simulator  training  across  a  spectrum  of 
Air  Force  Commands  and  a  review  of  the  state-of-the-art  capabilities  of  four 
systems  which  utilize  advanced  instructor  support  features.  The  review  of 
these  systems  included  both  data  collection  during  site  visits  and  the  review 
of  course  documents  collected  during  Phase  I.  During  each  site  visit,  ATD 
training  requirements,  including  aircrew  training  objectives,  simulator 
characteristics,  and  instructor  control  and  informational  requirements  were 
collected  and  assessed. 

Simulator  training  was  observed  at  the  following  MAJCOMs. 

1.  Tactiel  Air  Command  (TAC) :  A-10,  F-15,  F-16  aircraft 

2.  Military  Airlift  Command  (MAC):  C-130,  C-141  aircraft 

3.  Strategic  Air  Command  (SAC):  B-52,  KC-135  aircraft 

4.  Air  Training  Command  (ATC) :  T-37,  T-38  aircraft 

Locations  and  dates  of  these  visits  are  identified  in  Appendix  A.  The  interview 
guide  utilized  during  these  visits  is  included  in  Appendix  B. 
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Figure  1.  Approach  Roadmap 


Training  documentation  including  course  syllabi,  task  and  objectives 
documents,  instructor  guides,  and  simulator  manuals  was  collected.  Complete 
documentation  was  difficult  to  obtain.  Over  30  documents  were  reviewed 
for  information  regarding  aircrew  training  objectives,  simulator  character¬ 
istics,  and  instructor  control  and  informational  requirements.  A  listing  of 
course  documentation  acquired  and  reviewed  is  included  as  Appendix  C. 

Four  additional  training  systems  have  been  built  as  modifications  to 
existing  ATDs  and  were  included  in  this  review.  These  systems  were  designed 
to  specifically  provide  advanced  instructor  support  capabilities  and  include: 

1.  Automated  Flight  Training  System  (AFTsf®)  for  F-4E  and  A-7D  aircrews. 

2.  C-5A  Performance  Measurement  System  (C-5A  PMS) . 

3.  F-14  Instructor  Support  System  (F-14  ISS). 

4.  Air  Refueling  Part  Task  Trainer  Instructor  Support  System  (ARPTT 
ISS)  for  B-52  aircrews. 

A  brief  description  of  each  of  these  systems  is  provided  in  Appendix  D. 

Descriptive  data  on  these  systems  were  collected  and  reviewed.  These 
included  functional  specifications,  design  documents,  operation  manuals,  and 
program  source  listings.  Test  and  evaluation  data  were  also  reviewed.  Appendix 
E  provides  a  listing  of  the  documentation  reviewed. 

Interviews  were  conducted  with  personnel  who  were  directly  involved  in 
the  development  of  the  systems .  The  purpose  was  to  verify  the  information 
contained  in  the  documentation  and  to  derive  lessons  learned  regarding  system 
functions  and  hardware  and  software  implementation.  Visits  were  made  to 
selected  MAJCOM  sites  to  observe  training  and  to  determine  user  attitudes 
regarding  the  design  and  implementation  of  the  advanced  instructor  support 
features.  These  visits  are  included  in  the  Appendix  A  listing. 

The  third  phase  culminated  in  the  writing  and  production  of  the  actual 
guidelines,  including  a  sample  specification  and  procedure  by  which  the 
specification  was  generated. 

Data  management  was  a  major  concern  throughout  this  project.  Preliminary 
information  and  data  obtained  as  a  result  of  the  review  of  training  and 
systems  documentation  were  entered  into  computer-based  files.  These  files 
contained  data  describing  each  ATD,  the  aircraft,  instructor  support  features, 
tasks  trained,  performance  measurement,  and  scoring  and  information  sources. 
After  site  visit  information  was  obtained,  these  data  were  reworked  into  data 
files  which  describe  each  ATD  in  terms  of  training  objectives,  instructor/ 
operator  station  (IOS)  type,  I0S  control,  IOS  displays,  performance  evaluation, 
ISFs,  and  comments  regarding  lessons  learned. 

Project  notebooks  for  internal  use  were  utilized  to  organize  correspondence 
and  data  relating  to  the  project.  A  project  library  that  was  maintained 
included  over  100  references,  including  recent  research  publications  that 
address  the  issues  of  ISF  design  and  use.  These  sources  are  Identified  in 
Appendices  A  and  D  and  in  the  References  and  Bibliography  sections.  A  glossary 
of  ISF- related  terms  and  acronyms  were  compiled  and  updated  periodically 
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Investigation  of  the  ATD/Specif ication  Process 

Before  the  guidelines  content  could  be  developed,  an  investigation  of  the 
ATD  acquisition/specification  process  was  necessary  to  identify  the  type  of 
information  needed  by  SimSPO  in  order  to  properly  specify  ISS  requirements. 
This  investigation  also  provided  insight  about  Guidelines  user  needs  and 
helped  to  determine  where  the  Guidelines  would  apply  in  the  acquisition/specifi¬ 
cation  process. 

This  investigation  focused  specifically  on  the  part  of  the  acquisition/spe¬ 
cification  process  from  the  initial  identification  of  need  by  the  MAJCOM  user, 
to  the  communication  of  need  to  the  SimSPO  specification  writer,  and  to 
expression  of  these  requirements  in  the  actual  Prime  Item  Development  specifica¬ 
tion. 

Several  Air  Force  regulations  and  other  publications  that  describe  this 
process  were  reviewed.  These  sources  are  noted  in  the  References  and 
Bibliography.  A  fact-finding  meeting  was  held  with  SimSPO  and  MAJCOM  personnel 
in  order  to  document  the  current  specification  process  and  to  further  establish 
the  content  and  format  needs  of  the  guidelines  users.  Concept  papers  were 
prepared  prior  to  the  meeting  in  order  to  generate  discussion  and  obtain 
feedback  as  to  the  direction  of  the  study.  These  papers  addressed  such  topics 
as  state-of-the-art  instructor  support  capabilities,  automated  performance 
measurement  of  simulator  tasks,  system  hardware  and  software  design  and 
implementation  considerations,  alternative  guideline  formats,  and  the  ATD 
acquisition  process.  These  papers  proved  a  useful  communication  tool  in  that 
their  content,  as  well  as  the  feedback  obtained  from  meeting  participants, 
helped  shape  the  ultimate  content  and  format  of  the  guidelines . 

A  sampling  of  past  ATD  specifications  (including  those  for  the  A- 10,  F-15, 
F-16,  and  C-130  aircraft)  was  also  acquired  and  examined  to  gain  familiarity 
with  current  specification  practices  and  to  see  if  a  generic  computer  system 
architecture  for  the  support  of  ISFs  was  currently  applied  or  implied. 

Instructor  Support  Feature  (ISF)  Analysis 

An  analysis  compared  ISF  utilization  and  effectiveness  among  the  represen¬ 
tative  ATDs  and  for  the  four  systems  with  advanced  instructor  support  capabi¬ 
lities.  This  analysis  provided  data  for  the  Lessons  Learned  section  of  the 
ISF  definitions  included  in  Section  II  of  the  ISF  Guidelines. 


Task  Commonality  Analysis 


The  purpose  of  the  task  commonality  analysis  was  to  determine  whether  a 
commonality  exists  among  tasks  trained  at  the  ATDs  selected  as  representative 
from  the  four  MAJCOMs  and  the  four  systems  with  advanced  instructor  capa¬ 
bilities.  If  it  was  demonstrated  that  there  are  common  tasks  taught  in  most 
ATDs  and  that  the  four  systems  address  the  monitoring  of  these  tasks,  then  it 
could  be  reasoned  that  the  ISS  technology  which  has  already  been  developed 
could  be  applied  to  meet  current  and  future  ATD  training  requirements . 
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Because  five  of  the  nine  ATDs  investigated  conduct  pilot  training  only,  the 
scope  of  the  analysis  included  a  comparison  of  pilot  flight  station  tasks 
only.  A  detailed  review  of  training  documentation,  including  the  simulator 
training  syllabi  and  instructor  guides,  was  made  in  order  to  identify  tasks 
trained  on  each  ATD.  This  documentation  describes  the  general  training 
scenario  and  the  specific  training  objectives  for  each  event.  In  many  cases, 
however,  a  description  of  the  training  events  on  a  task-by-task  basis  is  not 
provided.  Extensive  interviews  with  instructors  were  conducted  to  obtain 
further  information  about  which  specific  tasks  were  monitored  during  simulator 
training  sessions. 

A  listing  of  training  tasks  by  phase  of  flight  was  compiled  for  each 
ATD .  The  tasks  were  then  grouped  by  type  into  the  following  categories : 
Normal  Flight,  Normal  Procedures,  Emergency  Flight,  Emergency  Procedures, 
Tactical  Flight,  and  Tactical  Procedures.  The  tasks  which  are  monitored  in  the 
four  systems  were  then  identified  from  existing  documentation  and  grouped 
into  the  previously  defined  task  listing.  To  facilitate  comparison  and 
analysis  of  these  training  tasks,  a  task-listing  matrix  was  generated;  this 
matrix  has  been  included  in  Appendix  E  of  the  ISF  Guidelines. 


The  four  systems  representative  of  the  current  capabilities  were  compared 
on  their  ability  to  monitor,  to  compute  performance  measures  and  score,  and 
to  trigger  other  instructional  support  actions.  Common  and  effective  functional 
characteristics  were  identified  to  provide  guidance  for  future  ISS  design. 
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The  ISF  requirements  of  these  four  systems  were  then  analyzed  from  a 
systems  engineering,  hardware,  and  software  perspective.  This  analysis 
provides  reference  for  future  implementations. 


Several  alternative  guideline  formats  were  considered  for  presentation  of 
the  guidelines  information.  A  literature  search  of  format  types  used  in 
other  design  guides/handbooks  was  conducted  so  that  an  effective  framework  for 
presentation  of  the  content  of  the  guidelines  could  be  designed.  Three  basic 
format  types  were  considered  and  are  briefly  described  as  follows: 

1.  Checklist .  This  format  type  is  used  in  the  AFSC  Design  Handbook 
(1977).  Checklists  are  provided  to  use  for  establishing  design  requirements 
and  for  verifying  that  the  requirements  have  been  met.  The  intent  is  to 
ensure  that  all  applicable  design  factors  have  been  examined  and  that  all 
problems  were  resolved  or  otherwise  determined  unimportant  to  the  design. 
Each  checklist  item  in  the  handbook  cross-references  to  another  section 
entitled  "Design  Notes"  which  provides  coverage  of  a  specific  topic. 

2.  Narrative .  The  particular  format  type  under  consideration  was  one 
used  in  Caro,  Pohlmann ,  and  Isley  (1979).  This  format  was  intended  to  communi¬ 
cate  information  on  instructional  features  to  engineering  and  other  simulator 
design  personnel.  It  consisted  of  the  following  elements: 
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Feature 

Definition 

Purpose  and  Intended  Use 
Function  Description 
Concurrent  Events  Description 
Feature  Diagram 

3.  Model  Specification  with  Accompanying  Handbook.  This  format  style 
was  used  in  Hritz  and  Purifoy  (1980).  The  accompanying  handbook  provides  a 
set  of  instructions  on  how  to  apply  the  model  specification  to  a  specific 
application.  For  each  paragraph  and  subparagraph  of  the  specification,  the 
following  sections  are  addressed  in  the  handbook: 

Rationale  and  Guidance 
Performance  Parameters 
Background  and  Sources 
Lessons  Learned 

Two  alternative  format  layouts  were  considered  for  the  Guidelines:  the 
standard  header/paragraph  layout  that  is  used  in  most  documents,  and  the 
Information  Mapping®  writing  style  described  in  Horn  (1983)  that  offers  a 
more  unique  visual  presentation.  The  Information  Mapping®  style  provides  a 
structured  format  using  labeled  blocks  to  organize  the  material.  These  labels 
highlight  the  structure  of  the  information,  making  it  easy  for  the  reader  to 
locate  relevant  detail.  Because  information  is  presented  in  modular  units, 
changes  and  updates  can  be  easily  accommodated. 

Guidelines  samples  were  prepared  in  these  two  alternative  layouts  and 
then  presented  to  IOS  Working  Group  members  who,  as  representative  of  Guidelines 
users,  selected  the  Information  Mapping®  style  as  the  preferred  layout. 


The  feasibility  of  on-line  computer  format  alternatives  for  ease  of  update 
was  also  considered.  Final  delivery  of  the  Guidelines  included  IBM  personnel 
computer  compatible  diskettes  that  contained  word  processing  files  and  software, 
in  addition  to  the  hardcopy. 

Guidelines  Development 

The  development  of  Guidelines  content  was  a  iterative  process.  During  the 
course  of  the  project,  its  content  outline  was  revised  several  times  to  meet 
the  needs  of  the  Guidelines  users.  The  document  is  organized  into  four 
sections;  each  section  is  intended  to  be  read  by  different  users  at  different 
times  in  the  ATD  procurement  process: 

I .  Overview 

II.  Instructor  Support  Features 

III.  Selecting  Instructor  Support  Features 

IV.  Providing  Operational  Information 
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Development  of  ISF  Definitions  (Section  II,  Instructor  Support  Features) 

Based  on  an  extensive  survey  of  research  publications  which  address  the 
subject  of  ISFs,  specifically  Caro,  Pohlmann,  &  Isley,  1979;  Semple,  Cotton,  & 
Sullivan,  1981;  Polzella,  1983;  Leaf,  Fitzpatrick,  &  Gunzburger,  1983;  and  the 
present  analyses,  sixteen  instructor  support  features  were  identified  for 
inclusion  in  Section  II  of  the  Guidelines.  Originally,  only  "advanced  instruc¬ 
tional  features"  (e.g.,  performance  measurement,  scenario  control)  were 
intended  to  be  addressed.  Advanced  instructional  features  are  those  features 
which  increase  the  instructor's  efficiency  and  effectiveness  by  reducing  the 
workload  and  providing  support  in  the  total  instructional  process  of  simulator 
training.  However,  it  was  felt  that  more  basic  ISFs  (e.g.,  record/replay, 
freeze)  should  also  be  included.  It  should  be  noted  that  the  term  "instructor 
support  feature  (ISF)"  was  specifically  selected  for  use  because  it  so  closely 
describes  the  purpose  of  these  features.  An  attempt  was  made  to  provide  a 
concise  definition  for  each  feature,  describe  its  instructional  value,  list 
additional  considerations  to  be  examined  when  fine-tuning  the  specification, 
note  related  features,  and  provide  examples  and  lessons  learned  based  on  past 
experience.  The  content  of  the  ISF  definitions  is  based  on  the  analyses 
results  and  a  review  of  the  research  material  cited. 

Drafts  of  the  definitions  were  discussed  at  a  working  meeting  held 
in  April  1985  at  Luke  AFB.  The  meeting  was  attended  by  personnel  from  the 
4444th  Operational  Squadron,  as  well  as  from  AFHRL  and  SimSPO.  Feedback  was 
sought  to  determine  whether  they  met  the  typical  user’s  needs.  Initial 
response  by  this  representative  group  of  operational  users  was  positive  and 
suggestions  for  improvement  were  incorporated  into  the  Guidelines. 


A  control  mechanism  successfully  implemented  in  the  four  systems  with 
advanced  instructor  support  capabilities  was  that  of  mapping  training  tasks 
into  modular  data  files  referred  to  as  "task  modules."  In  these  systems,  task 
modules  have  successfully  served  as  the  means  by  which  ISFs  are  implemented. 
Because  of  this  success,  the  task  module  concept  has  been  introduced  in  the 
Guidelines  (Section  IV,  Providing  Operational  Information)  as  one  approach  to 
implementing  a  data-driven  instructor  support  system. 

Task  modules  are  presented  as  tools  to  be  used  by  operational  users 
which  provide  a  framework  for  specifying  ISS  requirements  so  that  required 
training  support  functions  will  be  provided  by  the  ATD.  For  example,  task 
modules  identify  the  conditions  triggering  or  terminating  a  training  objective, 
and  define  the  appropriate  aircrew  performance  measurement  procedures  and 
information  displays  with  respect  to  that  objective.  Refinement  of  information 
contained  in  task  modules  continues  as  training  requirements  are  defined  more 
explicitly.  Ultimately,  this  information  is  translated  into  data  files  and 
modular  software  programs  by  contractors.  Thus  task  modules  are  the  bases  of 
an  approach  to  modular  software  architecture  from  which  an  ISS  could  be  built, 
which  would  control  the  operation  of  the  system. 
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ISF  Guidelines  Appendices 

The  following  appendices  were  also  developed  to  be  included  in  the  ISF 
Guidelines  document. 

Appendix  A.  Aircrew  Training  Documentation.  This  appendix  lists  the 
training  documents  and  syllabi  from  thirteen  aircrew  training  programs  which 
were  collected  and  reviewed. 

Appendix  B.  Bibliography.  This  appendix  lists  the  informational  literature 
relating  to  the  subject  of  instructor  support  features  which  was  reviewed. 

Appendix  C.  Sample  Specification.  To  illustrate  the  use  of  the  Guidelines, 
a  sample  specification  for  a  future  ATD  acquisition  of  interest  to  SimSPO  was 
developed.  This  provided  not  only  an  opportunity  to  validate  the  Guidelines 
but  also  an  opportunity  to  positively  affect  a  procurement  as  well.  In  addition, 
by  engaging  in  this  process,  a  procedure  for  analyzing  instructor  support 
requirements  for  future  ATD  procurements  was  derived. 

It  was  hoped  that  the  ATD  for  which  the  sample  specification  was  to  be 
generated  would  be  identified  during  Phase  I.  The  system  initially  designated 
by  the  Air  Force  was  the  C-130.  The  selection  was  then  changed  to  the  F-15E 
Dual  Role  Fighter  (DRF) .  Final  selection  of  the  F-16  upgrade  was  not  made 
until  the  end  of  Phase  II.  The  delay  in  the  identification  of  the  system  did, 
however,  have  a  positive  impact.  It  enabled  the  sample  specification  to 
serve  a  useful  purpose  by  making  it  a  working  document  that  would  potentially 
affect  an  actual  procurement,  rather  than  merely  provide  a  hypothetical  sample 
that  would  simply  demonstrate  the  application  of  the  Guidelines. 

Two  meetings  with  the  TAC  Instructional  Systems  Development  (ISD) 
Squadron's  F-16  training  program  upgrade  representatives  were  held  at  Luke  AFB 
in  order  to  identify  training  tasks  and  Instructor  support  system  require¬ 
ments.  Based  on  their  inputs,  a  listing  of  F-16  training  objectives  was 
compiled,  similar  in  format  to  the  descriptions  found  in  the  task  commonality 
matrices.  Identification  of  each  training  objective  enabled  further  definition 
of  requirements  in  terms  of  briefing,  initialization,  control,  instruction, 
monitoring,  and  debriefing.  Constructing  these  tasks  sequentially  into 
hypothetical  training  events ,  and  examining  them  in  the  context  of  a  total 
training  exercise,  enabled  instructor  support  requirements  to  be  defined  even 
further . 


Various  specifications  were  reviewed  for  content  and  format  structure  as 
well.  These  included  MIL-STD-490  (1968)  and  the  draft  specification  by  Leaf 
et  al.  (1983).  The  basic  format  structure  of  the  sample  specification  remains 
unchanged  from  previous  specifications;  the  content,  however,  is  entirely 
different.  Using  the  procedure  described  in  Section  III  of  the  Guidelines, 
the  sample  specification  was  generated.  Functional  definitions  of  the  required 
instructor  support  features,  written  by  SimSPO  staff  based  on  the  ISF  defi¬ 
nitions,  were  tailored  and  included  to  meet  the  needs  of  operational  users. 

Appendix  D. _ Training  Sites  Visited.  This  appendix  lists  the  data 

collection  trips  that  were  made  to  operational  ATD  and  prototype  ISS  sites. 
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Appendix  E.  Task  Commonality  Analysis.  The  matrices  contained  in  this 
appendix  provide  a  listing  of  general  tasks  currently  trained  at  nine  ATD 
sites.  Although  this  table  is  provided  to  show  commonality  of  tasks  across 
several  different  training  sites,  it  may  provide  guidance  in  the  development 
of  a  list  of  task  modules  for  future  ATDs. 
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Appendix  F.  ISS  Implementation  Considerations.  ISS  cost  and  implementation 
guidelines  have  been  presented  as  appendix  material  for  SimSPO  personnel  who 
have  technical  backgrounds  but  limited  experience  with  ISS  implementation. 

Appendix  G.  Sample  Task  Modules.  Representative  samples  of  specific  task 
modules  were  developed  to  provide  specific  cases  for  reference  by  those 
involved  in  the  specification  process. 


III.  RESULTS 


This  section  presents  the  findings  of  the  investigations  and  analyses 
efforts.  These  results  are  described  under  the  following  headings  and  are 
presented  in  the  same  order  as  described  in  Methods  (Section  II)  immediately 
above : 

1.  Current  ATD  Acquisition/Specification  Process 

2.  Instructor  Support  Feature  Analysis 

3.  Task  Commonality  Analysis 

4.  Internal  ISS  Analysis 

5.  Hardware/Software  Implications 

6.  Guidelines  Format  Selection 

7.  The  Final  Product  --  The  ISF  Guidelines 

Current  ATD  Acauisition/Speclf icatlon  Process 

Acquisition  of  ATDs  is  handled  by  the  Deputy  for  Simulators,  SimSPO,  of 
the  Aeronautical  Systems  Division,  Air  Force  Systems  Command  (AFSC/ASD/YW) . 
Sir.SPO  is  involved  in  specifications  for  acquisitions  that  range  from  training 
programs  to  products  and  equipment.  Some  of  these  acquisitions  have  included 
ISSs  for  ATDs.  The  SimSPO  follows  established  procedures  from  ATD  project 
inception,  through  contract  award,  and  to  final  transfer  of  the  ATD  to  the 
using  Command  and  the  Air  Force  Logistics  Command  (AFLC) .  These  procedures 
are  mandated  by  Air  Force  Regulation  (AFR)  800-2  (1982),  Acquisition  Program 
Management . 

The  SimSPO,  however,  is  not  always  involved  with  acquisitions.  For 
example,  simulator  "refurbishments , "  which  may  include  changes  to  the  simulator, 
can  be  procured  through  the  AFLC.  AFLC,  with  Ogden  Air  Logistics  Center 
acting  as  the  implementing  agency,  is  responsible  for  simulator  modifications 
and  maintenance  after  the  initial  acquisition.  The  regulation  governing 
Modification  Program  Approval  and  Management  is  AFR  57-4  (1983). 

Need  Identification 


Training  requirements  are  initially  identified  by  the  MAJCOMs  using  the 
ISD  process,  an  approach  to  the  analysis  of  training  requirements  and  develop¬ 
ment  of  training  systems.  This  includes  performance  of  a  task  analysis  of  the 
missions  to  be  trained  and  media  selection.  Relevant  regulations  include 
AFR  50-8  which  requires  that  the  ISD  process  be  utilized  in  the  identification 
of  training  requirements,  and  AFR  50-11  (1977)  which  requires  that  all  training 
equipment  be  designed  according  to  ISD  methodology. 

Acquisition  of  a  new  ATD  is  formally  initiated  by  a  Statement  of 
Operational  Need  (SON),  a  statement  of  training  requirements  generated  by  one 
of  the  MAJCOMs.  It  is  a  formal  document  which  identifies  an  operational 
deficiency  and  states  the  need  for  a  new  or  improved  capability.  In  a  SON  for 


acquisition  of  a  new  weapon  system,  the  statement  of  requirements  may  be 
stated  very  generally,  usually  amounting  to  a  single  statement  that  an  ATD 
will  be  required.  A  SON  specific  to  the  ATD,  on  the  other  hand,  usually 
provides  more  substantial  detail,  such  as  operator  control  and  ISF  requirements. 
Therefore,  the  level  of  detail  varies  substantially  among  SONs .  In  the  case 
of  the  acquisition  of  a  major  system,  and  when  approval  by  the  Secretary  of 
the  Air  Force  is  required,  the  preparation  of  a  Mission  Element  Needs  Statement 
(MENS)  is  necessary.  The  Air  Force  Regulation  which  addresses  the  preparation 
of  the  SON  and  MENS  is  AFR  57-1,  Statement  of  Operational  Need  (SON). 

Concept  Development 

The  using  Command  issues  a  draft  SON  document  and  distributes  it  to  the 
other  implementing  and  participating  Commands  (e.g.,  AFLC,  AFSC,  HQ  USAF  and 
ATC)  for  comment.  These  Commands  in  turn  contribute  data  and  experience  on 
the  technical  base,  logistics  costs,  human  factors,  training,  etc.  The  using 
Command  notes  their  input  to  refine  and  update  the  SON.  During  this  time, 
SimSPO  and  the  using  Command  begin  to  work  together  to  define  user  needs  more 
specifically.  After  suggested  changes  are  incorporated  and  cost  estimates 
included,  the  final  version  of  the  SON  is  sent  to  HQ  USAF  for  evaluation  and 
validation. 

HQ  USAF  assesses  the  technology  and  constraints  and  identifies  the 
estimated  resources  to  satisfy  the  need  and  requests  review  by  the  other 
Commands  so  a  recommended  course  of  action  can  be  provided  as  well  as  a  deter¬ 
mination  of  priority  ranking  for  the  SON  or  MENS.  The  issuance  of  the  Program 
Management  Directive  (PMD)  by  HQ  USAF  notifies  all  concerned  as  to  whether  the 
SON  or  MENS  has  been  validated  or  approved,  in  whole  or  in  part,  or  dis¬ 
approved.  It  is  at  this  point,  upon  issuance  of  the  PMD  and  AFSC  Form  56, 
that  the  program  is  assigned  to  SimSPO.  The  SimSPO  assigns  a  Program  Manager 
(PM)  whose  role  is  to  guide  the  program  toward  achieving  the  program  objec¬ 
tives.  The  PM  prepares  a  Program  Management  Plan  (PMP)  that  lays  out  the 
acquisition  strategy  and  defines  the  support  requirements  of  the  participating 
organizations.  The  PMP  is  submitted  with  the  revised  PMD  to  influence  the 
directives . 


The  PMD  is  the  official  management  directive  that  provides  direction  to 
the  implementing  and  participating  commands  and  authorizes  the  commitment  of 
resources  to  satisfy  the  operational  need.  As  directed  by  the  PMD,  the  SimSPO 
works  with  subject  matter  experts  from  the  using  command  to  fully  evaluate  the 
operational  and  supporting  implications  of  various  alternative  design  ap¬ 
proaches.  The  SimSPO  relies  heavily  on  the  operating,  supporting,  Operational 
Test  and  Evaluation  (OT&E)  and  other  participating  commands  to  participate 
actively  in  the  acquisition  life  cycle.  All  participants  coordinate  to  ensure 
system  design,  operational,  and  support  concept  development. 

Advisory  resources  are  available  to  the  SimSPO  as  well.  The  ASD  engi¬ 
neering  directorates  provide  experienced  engineering  personnel  who  perform 
front-end  analysis  and  rough  costing  support.  Training  equipment  specialists 
participate  early  in  the  definition  phase  through  the  validation  of  device 
functional  requirements.  From  the  Air  Force  Human  Resources  Laboratory, 


training  psychologists  and  engineering  personnel  specializing  in  simulation 
research  may  make  contributions  during  all  phases  of  the  acquisition  process. 

Generation  of  Specification 

A  team  composed  of  personnel  from  SimSPO  and  also  operational  training, 
management  and  engineering  personnel  from  the  using  Command  is  tasked  with 
developing  the  actual  ATD  Prime  Item  Development  Specification  (PIDS).  SimSPO 
personnel  meet  with  user  representatives  via  site  visits  to  determine  how  the 
system  will  be  used,  facilities  requirements,  etc.  A  draft  specification  is 
prepared  in  accordance  with  MIL-STD-490.  This  draft  is  reviewed  in  detail  by 
the  users  as  well  as  by  the  Director  of  Requirements  (DR)  and  Director  of 
Operations  (DO)  at  the  MAJCOM.  In  some  cases,  a  draft  of  the  specification  is 
sent  to  industry  for  comment.  After  modifications  have  been  made,  the  final 
version  of  the  specification  is  published. 

Problem  Areas  Identified 

The  investigation  of  the  ATD  acquisition/specification  process  and 
discussion  with  MAJCOM  and  SIMSPO  personnel  pointed  to  several  problem  areas 
relating  to  ATD  acquisition.  Identification  of  these  problems  enabled  the 
definition  of  a  guidelines  content  that  would  meet  user  requirements. 

Because  equipment  and  training  requirements  are  often  developed  con¬ 
currently,  front-end  analysis  does  not  always  precede  procurement.  Typically, 
the  simulator  is  procured  first,  and  then  the  training  is  defined.  ISD  goes  on 
during  the  acquisition  process  but,  unfortunately,  does  not  always  drive  the 
requirements.  This  has  led  to  the  design  and  implementation  of  systems  that 
often  do  not  support  the  intended  training  objectives  or  meet  instructor  needs. 
If,  Instead,  the  operational  users  would  clearly  identify  their  requirements 
and  communicate  them  to  the  specification  writers  at  the  outset,  specifica¬ 
tions  could  be  generated  that  would  more  accurately  reflect  their  needs. 

In  an  effort  to  reverse  the  process  of  procuring  simulators  first  and  then 
defining  training,  the  SimSPO  is  attempting  to  move  towards  contractor -provided 
training  programs.  In  the  future,  a  contractor  may  be  responsible  for  the 
entire  process,  including  the  front-end  analysis,  development  of  necessary 
media,  and  the  operation  and  maintenance  of  the  resulting  program.  The  SimSPO's 
role  will  be  to  oversee  the  process.  This  trend  has  significant  implications 
to  the  development  of  ISSs  in  ATDs.  Specific  guidelines  for  specifying 
requirements  based  on  past  procurements  would  provide  a  repository  for  lessons 
learned  to  benefit  all  future  participants  and  would  promote  sharing  of 
information. 

Another  problem  area  is  the  lack  of  personnel  who  are  properly  trained  in 
identifying  ATD  training  and  ISF  requirements.  This  is  partly  due  to  personnel 
turnover,  unfamiliarity  with  state-of-the-art  technology,  and  the  lack  of 
adequate  guidelines  for  selecting  ISFs  of  new  procurements.  Unfortunately, 
what  happens  too  often  is  that  an  ATD  procurement  is  based  on  the  dictates  of 
an  individual,  who  is  only  familiar  with  one  particular  device,  rather  than  a 
result  of  an  effort  based  on  "corporate"  knowledge.  In  other  cases,  there  is  a 
tendency  to  over-acquire  "just  in  case.”  The  resulting  ATDs,  then,  are  either 
insufficiently  equipped  with  features  or  are  too  complex  to  utilize  fully. 
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It  was  also  noted  that  the  terminology  used  to  define  ISFs  is  used  very 
loosely  and,  in  some  cases,  leads  to  the  design  and  development  of  features 
that  do  not  fulfill  user  requirements.  At  times,  there  is  a  basic  misunder¬ 
standing  as  to  what  these  features  are  to  provide ,  what  they  are  to  be  used 
for,  and  what  benefits  are  to  be  gained.  It  is  important,  therefore,  that 
requirements  personnel,  specification  writers,  users,  and  contractors  alike  be 
thoroughly  familiar  with  the  terminology  used  in  defining  ISFs  and  have  an 
understanding  as  to  each  feature's  purpose  and  intended  use. 

For  the  most  part,  previous  specifications  have  not  provided  enough 
direction  to  the  contractor  in  regards  to  user  training  and  instructor  support 
feature  requirements.  In  many  cases,  the  description  of  required  capabilities 
is  unclear.  This  has  resulted  in  the  design  and  implementation  of  ATDs  that 
have  not  completely  satisfied  user  needs.  For  example,  specifications  that 
simply  list  desired  ATD  functions  without  regard  to  how  an  instructor  must 
exercise  those  functions  and  without  regard  to  their  purpose  and  intended  use 
may  produce  a  "feature -rich"  ATD  with  unusable  features. 

ISF  Guidelines  Intended  Audience  and  Use 

As  a  result  of  this  investigation  of  the  ATD  acquisition  process,  it 
became  clear  that  a  vehicle  for  providing  common  vocabulary  for  the  communi¬ 
cation  of  instructor  support  requirements  would  alleviate  some  of  these 
problems.  By  standardizing  the  terminology  used  in  describing  ISS  requirements, 
the  Guidelines  would  facilitate  communication  among  the  MAJCOM  personnel  who 
identify  initial  training  requirements  and  needs,  the  SimSPO  specification 
writers/procurers  who  are  tasked  with  specifying  requirements  in  the  PIDS,  and 
the  contractors  who  build  ATDs. 

The  ISF  Guidelines  provide  procedures  for  analyzing  training  and  instruc¬ 
tor  support  requirements  to  specify  ISSs,  provide  descriptions  of  ISFs,  and 
educate  the  reader  about  current  system  capabilities.  These  Guidelines  can 
assist  MAJCOM  operational  personnel  in  identifying  their  requirements  and  help 
SimSPO  personnel  express  those  requirements  accurately  in  procurement  specifica¬ 
tions  . 

Figure  2  illustrates  the  anticipated  points,  within  the  acquisition 
process,  where  the  Guidelines  are  expected  to  be  utilized.  As  a  guide  to  the 
SON,  the  Guidelines  would  provide  assistance  to  operational  personnel  in 
selecting  ISFs  to  include  in  a  future  procurement.  As  a  guide  to  the  PIDS, 
they  would  provide  a  functional  description  of  ISFs  in  specification  termi¬ 
nology.  Finally,  during  Prime  Item  Development,  the  Guidelines  would  provide 
operational  personnel  with  assistance  in  developing  the  task  modules  that 
would  ultimately  be  used  by  the  system  developers. 


Instructor  Support  Feature  Analysis  Results 

The  purpose  of  the  ISF  analysis  was  to  compare  ISF  utilization  and 
effectiveness  for  the  representative  MAJCOM  ATDs  and  the  four  systems  with 
advanced  ISFs.  Results  of  the  data  gathering  and  site  visits  are  described  in 
this  section.  Tables  1  and  2  contain  supplementary  information.  Refer  to 
Section  II  of  the  ISF  Guidelines  for  complete  definitions  of  each  ISF  and 


Figure  2.  ISF  Guidelines  Use  in  the  AID  Acquisition  Process. 
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AUTOMATE 0  Run  score*  after  each  run.  these  Parameters  and  maintenance  of  Scores  task  modules  in  realtime.  Scores  task  modules  proficient/nonproficient 

PERFORMANCE  calculated  into  single  score  parameters  values  within  windows  Can  make  realtime  change.  Total  Evaluation  Template  {minimum 

MEASUREMENT  Adaptive  scheduling.  User  can  change  Crew  member  assessment  grade  calculated:  instructor  can  proficiency  and  actual) 

predefined  scoring  values  Crew  coordination  assessment  override  grade  Instructor  can  override  grade 

Mission  assessment 


The  following  is  a  summary  by  feature. 


\ 


Scenario  Control.  The  systems  which  had  fully  automated  scenario  control 
included  the  B-52  Weapon  Systems  Trainer  (WST) ,  the  F-16  Operational  Flight 
Trainer  (OFT),  C-130  OFT/WST,  and  the  four  systems.  One  of  the  differences  in 
the  operation  of  these  systems  is  how  the  missions  are  generated.  For  the 
ATDs,  generation  of  mission  scenarios  requires  users  to  have  data  entry  and 
programming  skills.  For  the  four  systems  that  have  advanced  ISFs,  the  training 
objectives  have  been  preprogrammed  into  a  database  and  allow  for  easy  selection 
of  preprogrammed  scenarios  by  the  instructor  during  the  briefing  or  the 
initialization  process. 

Realtime  Simulation  Variables  Control.  It  was  observed  that  the  manual 
selection  is  best  suited  for  informal  training  (e.g.,  continuation  training, 
instructorless  practice,  and  review).  Simulation  variables  were  available  on 
all  of  the  systems  visited.  The  amount  of  usage  depended  on  the  accessibility 
of  the  variable  and  whether  a  device  technician  was  available  during  training. 

The  selection  of  variables  by  re- initializing  the  simulator  seemed  to  break 
the  flow  of  training  and  detracted  from  the  realism  of  flight.  It  was  used 
most  for  instrument  navigation  training  where  approaches  to  different  airfields 
are  practiced. 

Variables  are  preprogrammed  in  the  B-52  WST  when  operating  in  the  "mission 
mode."  This  system  also  provides  for  manual  control  of  certain  variables  and 
allows  flexible  instructor  interaction. 

Malfunction  Control .  The  automatic  feature  is  rarely  used.  The  main 
reason  given  was  that  it  did  not  allow  the  instructor  to  tailor  training  to 
student  response  and  needs.  In  some  cases,  the  malfunction  activation  and 
deletion  did  not  correspond  to  the  desired  scenario.  In  particular,  the 
A-10's  preprogrammed  malfunctions  do  not  correspond  to  the  syllabus  being 
trained. 

The  B-52  ARPTT  and  the  F-14  ISS  provide  a  feature  whereby  the  instructor 
may  select  a  set  of  malfunctions  during  the  initialization  process.  These 
malfunctions  are  then  grouped  together  on  a  special  menu  and  may  be  readily 
accessed  during  the  exercise  for  actuation  and  deletion.  This  feature  is 
being  used  on  the  B-52  ARPTT  ISS. 

Reposition.  Repositioning  the  simulator  to  a  specific  location  was  used 
on  all  devices  visited  and  was  mostly  used  for  repetitive  training  (e.g., 
approaches) .  This  feature  is  accomplished  in  many  different  ways  and  in  varying 
degrees.  The  most  versatile  method  was  seen  on  the  A-10  ATD  where  the  simulator 
can  be  positioned  anywhere  within  the  active  geographic  graphics  display  by 
identifying  the  position  with  a  light  pen.  Repositioning  in  the  A-10  simulator 
may  also  be  accomplished  by  bearing  and  distance  from  a  fix,  latitude/longitude, 
or  by  identifying  a  previous  position  by  a  "snapshot  Initial  Condition  (I.C.)". 
The  most  common  way  to  reposition  was  accomplished  via  an  I.C.  reset.  The 
A-10  may  be  over-designed  for  the  training  requirement,  however.  The  I.C.  reset 
may  be  somewhat  restrictive,  time  consuming,  and  difficult  to  access. 


The  B-52  ARPTT  and  F-14  ISS  have  a  feature  whereby  the  trainer  may  be 
repositioned  to  the  beginning  of  any  flight- training  objective  (e  g.  pre-contact 
position,  initial  approach  fix).  This  feature  is  currently  used  in  the  ARPTT. 

The  repositioning  feature  on  the  F-14  OFT  was  somewhat  user  unfriendly  in 
that  repositioning  to  the  end  of  a  runway  would  cause  a  crash  condition  if  the 
landing  gear  were  not  down. 

IQS  Display  Control  and  Formatting.  Selection  of  displays  was  observed 
to  be  both  by  instructor  selection  or  by  automated  actuation.  In  most  cases, 
where  the  aircraft  was  geographically  referenced  on  a  display,  an  automated 
feature  would  provide  the  correct  reference.  For  example,  in  all  cases  when  the 
simulator  was  repositioned  to  the  beginning  of  an  approach,  an  approach 
display  would  automatically  come  up.  In  all  cases  when  a  geographic  plot  was 
being  displayed  and  when  the  aircraft  flight  path  approached  the  edge,  the 
display  would  change  to  the  next  appropriate  display. 

With  respect  to  displays  other  than  geographic  referencing,  the  instructor 
has  to  manually  select  any  display  associated  with  procedures  and  or  cockpit 
activity.  In  some  cases,  the  cockpit  controls  were  separated  by  relative 
position  in  the  cockpit.  In  other  cases,  display  of  controls  was  by  aircraft 
system.  This  separation  does  not  necessarily  provide  total  feedback  with 
respect  to  certain  procedures  so  instructors  had  to  use  some  other  "work 
around"  method  to  evaluate  performance.  An  exception  to  this  is  the  ISS 
systems  that  automatically  select  displays  appropriate  to  the  current  training 
objective. 

Those  systems  that  provided  checklists  and  procedures  on  special  displays 
were  not  often  used  because  the  checklists  and  procedures  were  outdated. 

Total  Svatem  Freeze.  All  systems  observed  have  this  feature,  and  it  was 
used  in  varying  degrees  depending  on  the  type  of  training.  At  the  Undergraduate 
Pilot  Training  (UPT)  level,  freeze  was  used  extensively  by  the  instructor  while 
providing  direct  feedback  and  corrective  action.  It  is  rarely  used  in  total 
mission  training. 

The  Crash  Override  feature  was  found  on  all  devices  and,  in  all  cases 
observed,  was  always  left  on. 

Partial  Freeze.  Partial/parameter  freeze  was  found  on  many  of  the 
devices  but  was  only  used  at  the  UPT  level.  Instructors  expressed  that  there 
is  little  training  value  for  this  feature  at  the  MAC/SAC/TAC  sites. 

Automated  Simulator  Demonstration.  This  feature  was  found  on  many  of  the 
devices  but  was  only  used  at  the  UPT  level,  and  then  not  very  often.  Instruc¬ 
tors  expressed  that  this  feature  may  have  some  value ,  but  that  they  would 
rather  use  the  simulator  time  for  "hands-on"  training. 

Simulator  Record/Reolav.  This  feature  was  found  on  many  of  the  devices  but 
was  mostly  used  at  the  UPT  level  and  then  only  by  certain  instructors.  Other 
instructors  expressed  that  this  feature  may  have  some  value,  but  again,  they 
would  rather  use  the  simulator  time  for  "hands-on"  training.  This  was  espec¬ 
ially  true  at  the  MAC/SAC/TAC  sites. 
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Hardcopv/Printout .  This  feature  was  found  on  most  devices.  Some  of  the 
systems  required  that  the  system  be  taken  down  from  the  runtime  programs 
prior  to  providing  the  hardcopy  output.  This  was  observed  to  be  restrictive 
to  actual  training,  and  by  the  time  the  copy  is  made,  the  instructor  has 
already  debriefed  the  student.  Some  of  the  devices  required  that  the  display 
being  hardcopied  be  frozen  while  the  output  was  produced.  This  may  be  disrup¬ 
tive  with  respect  to  real-time  feedback.  In  most  cases,  this  feature  was  not 
used  often  by  instructors.  The  reasons  varied  from  not  knowing  that  it 
existed  to  not  knowing  what  to  use  it  for. 

Procedures  Monitoring.  Some  of  the  systems,  including  all  those  that 
utilize  advanced  ISFs,  have  this  feature.  However,  it  is  not  used  much 
because  the  procedures  are  quickly  outdated  due  to  the  many  changes  (e.g., 
aircraft  configuration,  local  course  rules,  and  ATC  procedures). 

Automated  Performance  Measurement.  Some  of  the  systems  have  a  parameters 
monitoring  feature,  but  it  is  rarely  used.  Among  the  reasons  given  were  that 
it  is  to  difficult  to  set  up  and  that  the  results  have  little  relevance  to 
the  objective  being  evaluated. 

Some  of  the  WSTs  have  a  feature  called  automated  performance  measurement 
where  bomb  drops  and  missile  shots  are  scored.  This  feature  is  not  used  in 
the  A- 10  nor  F-16  because  instructors  feel  that  the  basic  simulation  does  not 
provide  the  cues  necessary  to  properly  launch  the  weapon.  The  F-15  missile 
scoring  is  used  for  basic  intercept  procedures  and  shots  made  beyond  visual 
range  (BVR) . 

The  four  advanced  instructional  systems  have  a  more  comprehensive  automated 
performance  measurement  feature  that  evaluates  performance  by  training  objec¬ 
tive.  The  B-52  ARPTT  ISS  has  just  recently  installed  this  feature;  however, 
there  has  not  been  enough  feedback  with  respect  to  operational  usage.  The 
AFTS®  scoring  has  either  been  used  or  not  used  depending  on  Command  support 
and  physical  location  (e.g.,  GCAs  are  used  frequently  in  Alaska  where  the 
weather  is  bad  and  pilot  proficiency  in  instrument  flying  is  critical  to 
safety  of  flight;  GCAs  are  not  used  very  much  at  Davis- Monthan  where  there 
may  be  1  day  in  the  year  when  an  instrument  approach  is  required  and  probably 
never  to  the  field  minimums). 

The  C-5A  PMS  and  F-14  ISS  were  designed  primarily  for  R&D  purposes,  and  no 
operational  data  were  evaluated  in  this  effort. 

Data  Storage  and  Analysis.  The  C-5A  PMS  has  the  capability  to  store  and 
analyze  scoring  data.  Operational  evaluation  data  are  not  yet  available. 
Because  this  feature  was  only  recently  installed  on  the  B-52  ARPTT  ISS,  no  data 
have  been  collected  with  respect  to  operational  usage. 


Remote  Graphics  Replay.  Systems  which  provide  this  feature  are  the  F-14 
and  B-52  ARPTT  ISSs .  The  F-14  ISS  was  an  R&D  development  system  and  the  opera¬ 
tional  usage  was  minimal.  The  ARPTT  Briefing/Debriefing  console  has  just 
recently  been  installed.  However,  the  operational  feedback  thus  far  has  been 
very  enthusiastic  with  respect  to  the  remote  graphics  replay  capability. 


(3) 

The  AFTS^also  provided  for  viewing  of  replay  of  exercise  geometry  and  con¬ 
troller  voice  messages  for  pre-mission  briefing  or  post-training  critique 
purposes  in  an  adjacent  briefing  area. 

Tutorial .  The  only  system  which  has  this  feature  is  the  F-14  ISS.  Not 
enough  data  were  collected  from  either  the  user  or  from  a  research  perspective 
because  this  feature  was  installed  just  prior  to  re-hosting  the  main  computer 
of  the  simulator.  Unfortunately,  this  re-configuration  made  the  ISS  inoper¬ 
ative. 

The  B-52  ARPTT  has  a  HELP  function  which  provides  the  user  with  system 
usage  information  which  may  be  accessed  upon  request  during  briefing  or 
debriefing.  The  C-5A  PMS  supports  HELP  pages.  Not  enough  information  has 
been  gathered  to  make  any  comments  on  this  feature. 


Conclusions 

A  Scenario  Control  feature  would  be  of  value  to  all  of  the  devices 
visited  in  that  it  would  reduce  the  instructor  workload  during  instruction.  A 
fully  automated  Scenario  Control  feature  (programmed  mission  scenarios)  would 
be  of  greater  value  to  long  sessions,  as  in  MAC/SAC,  and  to  a  lesser  extent  in 
ATC  where  a  console  operator,  because  of  the  design  of  the  system,  would 
provide  the  support.  Programmed  mission  scenarios  are  most  appropriate  for 
evaluating  progress  (e.g.,  checkrides)  where  standardization  is  important  or 
for  total  mission  training  practice,  but  during  normal  training  sessions,  they 
do  not  allow  for  instructor  flexibility  and  therefore  limit  training  effective¬ 
ness. 


During  many  of  the  training  events,  the  instructor  should  have  the 
capability  to  tailor  training  to  the  needs  of  the  individual  student.  Semi- 
automated  scenario  control  allows  the  instructor  to  create  a  tailored  mission 
to  meet  student  needs  and  provides  support  to  aid  the  instructor  during 
training.  Flexibility  (e.g.,  instructor  interaction  with  the  system)  during 
training  is  also  essential.  The  instructor  should  be  able  to  modify  variables, 
insert  and  remove  malfunctions,  and  move  forward  or  backward  in  the  profile 
to  satisfy  basic  instruction  and  student  progress. 

The  instructor  wants  flexibility  and  therefore  resists  automated  mal¬ 
functions  that  cannot  be  changed  and  automated  performance  measurement  that  is 
rudimentary  or  inaccurate.  Unfortunately,  the  operational  deficiencies  of 
some  of  these  features  have  alienated  instructors.  Improvements  in  ISF  design 
and  instructor  education  as  to  their  purpose  and  intended  use  should  lead 
toward  instructor  acceptance. 

Software  in  the  simulator  should  be  up-to-date  with  respect  to  the 
aircraft.  Data  relating  to  aircraft  procedures,  for  example,  must  be  easily 
modifiable  by  an  on-site  maintenance  activity  if  the  procedures  monitoring 
feature  is  going  to  be  utilized  and  appreciated  by  instructors. 

Most  of  the  ATDs  reviewed  provide  automated  performance  measurement  of 
simulator  variables  and  display  raw  performance  data.  The  ISSs  provide 
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automated  performance  measurement  by  training  objectives.  The  focus  on 
measurement  of  objectives  rather  than  parameters  was  observed  to  provide 
more  meaningful  information.  A  score  override  feature  would  help  diminish 
instructor  resistance  to  the  idea  of  automated  performance  measurement. 

Because  of  other  demands  on  instructor  time,  months  can  pass  without  any 
time  on  the  ATD,  making  it  difficult  for  the  instructor  to  maintain  his 
skills.  A  user-friendly  interface  design  that  allows  the  instructor  to 
operate  the  system  with  minimal  training  or  a  tutorial  built  in  the  system  for 
refresher  training  would  help  solve  this  problem. 

Instruction  on  utilization  of  ISFs  is  usually  informal  and  on  the  job. 
While  discussing  ISFs  with  instructors ,  it  was  noted  that  many  of  them  were  not 
aware  that  certain  features  existed.  In  some  cases,  they  did  not  know  how  to 
operate  them,  and  in  other  cases,  they  did  not  know  how  the  feature  could  be 
applied  to  training.  Some  instructors  viewed  the  simulator  as  a  substitute 
for  the  aircraft  on  the  flight  line  only  and  do  not  take  advantage  of  the  ATD 
as  part  of  a  total  training  system.  The  maximum  potential  of  ATDs  will  only 
be  attained  when  instructors  are  provided  with  the  proper  training  in  the 
usage  of  the  simulator  and  its  instructor  support  features. 

Task  Commonality  Analysis  Results 

The  task- listing  matrix  that  was  generated  for  the  task  commonality 
analysis  presents  a  listing  of  tasks  that  are  trained  on  the  ATDs  investigated 
and  those  incorporated  in  the  four  systems  with  advanced  ISFs.  This  matrix 
has  been  included  in  Appendix  E  of  the  ISF  Guidelines. 

A  strong  commonality  was  seen  among  simulator  training  tasks  grouped  in 
the  normal  flight,  normal  procedures,  emergency  flight,  and  emergency 
procedures  categories,  and  this  commonality  is  consistent  across  the  Air  Force 
MAJCOMs.  This  is  not  surprising,  since  these  types  of  tasks  reflect  the 
objective  of  the  basic  flight  training  philosophy,  which  includes  ensuring  that 
the  student  has  a  firm  understanding  of  procedures  and  limitations  of  the 
aircraft  and  can  demonstrate  this  knowledge,  as  well  as  the  motor  skill 
ability,  to  safely  operate  the  aircraft  under  normal  and  abnormal  conditions 
prior  to  the  first  flight. 

The  task-listing  matrix  indicates  that  conversion  training-task  require¬ 
ments  encompass  all  UPT  training- task  requirements.  In  conversion  training, 
the  student  is  familiarized  with  the  new  systems  and  utilizes  them  to  perform 
the  same  kinds  of  tasks  that  are  covered  in  UPT.  Primary  emphasis  is  on 
safety  of  flight  and  on  the  student's  ability  to  safely  operate  the  system 
within  the  procedures  and  guidelines  set  forth.  This  includes  starts,  takeoffs, 
landings,  instrument  and  basic  airwork  skills,  navigation,  use  of  checklists 
and  abnormal  situations.  Once  this  performance  has  been  demonstrated,  the 
student  is  introduced  to  basic  tactical  skills. 

Because  most  tasks  that  fall  into  the  categories  of  tactical  flight  and 
tactical  procedures  are  unique  to  each  tactical  mission,  a  strong  commonality 
among  training  tasks  was  not  observed.  Some  common  tasks,  however,  are 
applicable  to  all  major  operational  commands.  Such  mission- related  tasks  are 
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encompassed  in  Air-to-Ground  Attack  and  Electronic  Warfare.  These  tasks,  as 
taught  in  conversion  training,  provide  the  basic  foundation  for  continuation 
training  in  the  operational  squadrons.  The  AFTS®  for  the  A-7D  and  F-4E  is 
the  only  system  designed  to  monitor  tactical  training.  These  systems  provide 
air-to-air  and  air-to-ground  training  that  could  meet  the  needs  of  conversion 
training  in  these  areas. 

Nearly  all  conversion  training  tasks  in  the  first  four  task  categories 
(normal  flight,  normal  procedures,  emergency  flight,  and  emergency  procedures) 
have  already  been  incorporated  into  the  four  systems.  Tasks  which  have  not 
been  identified  as  ones  monitored  by  an  ISS  were  not  done  so  by  design.  The 
ISS  systems  for  the  F-14,  B-52  ARPTT,  and  C-5A  could  be  easily  modified  to 
monitor  any  additional  conversion  training- task  requirements. 

Internal  ISS  Analysis  Results 

This  section  describes  the  capabilities  that  allow  systems  to  monitor, 
compute  performance  measures  and  score,  and  trigger  other  instructional  support 
actions.  Refer  to  Appendix  F  for  a  detailed  comparison  of  ISS  capabilities 
of  the  four  systems  which  feature  advanced  instructor  support.  It  should  be 
noted  that  these  systems  were  all  modifications  to  existing  ATDs  and  that  the 
system  design  was  dictated  by  what  existed. 

Conceptual  View 

A  conceptual  view  of  the  ISS  is  presented  in  Figure  3.  The  ISS  is 
viewed  as  a  device  that  may  monitor  any  variable  present  in  the  simulation 
(e.g.,  variables  for  flight,  navigation,  controls,  environment)  and  take 
action  when  specified  criteria  are  met  (e.g.,  altitude  -  1000  ft).  The 
variables  to  be  monitored  and  the  actions  to  be  taken  are  dictated  by  a 
stored  script;  also,  a  new  script  can  be  initiated  when  specified  criteria  are 
met.  In  this  view,  the  ISS  is  a  controller  of  the  Instructional  process,  and 
much  depends  on  its  ability  to  monitor,  interpret,  and  diagnose  ongoing  perfor¬ 
mance  and  to  take  action  based  on  complex  statements  of  performance  conditions. 

Task  Module  Definition.  The  concept  of  a  task  module  is  used  in  two  of  the 
four  systems  examined  in  this  analysis  (F-14  ISS  and  ARPTT);  however,  it  has 
been  applied  to  the  entire  discussion,  since  it  provides  a  functional 
description  that  includes  training  relevance  and  some  independence  of  specific 
methods  of  engineering  implementation.  Task  modules  are  instructional  building 
blocks  that  describe  the  training  objectives  at  a  level  which  has  meaning  to 
instructors  and  that  can  be  used  by  the  machine  to  monitor  and  control  in  a 
manner  appropriate  to  the  instructional  objectives.  Thus,  if  a  training 
objective  is  to  execute  a  standard  instrument  departure  with  malfunctions 
inserted  under  specified  conditions  and  to  measure  flight,  navigational,  and 
procedural  performance,  all  such  information  would  be  included  in  the  task 
module  definitions  corresponding  to  the  training  objectives.  Upon  completion, 
new  task  modules  can  be  referenced  by  the  ISS  until  all  objectives  for  a  given 
mission  are  included.  The  task  module,  then,  is  a  control  file  or  a  script 
which  drives  the  ISS  and  which  can  be  interpreted  by  instructional  personnel 
and  related  to  training  requirements . 
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Figure  3.  Conceptual  Model  for  ISS  Control  and  Processing. 


Start  and  Stop  of  Computations.  An  important  feature  of  the  ISS  is  the 
ability  to  recognize  conditions  and  to  start  and  stop  ISS  processes;  for 
example,  to  start  a  new  task  module  when  a  complex  combination  of  events 
occurs  and  then  to  end  it  later  and  to  start/stop  measuring  performance 
(e.g.,  measure  average  heading  error  when  between  two  NAVAIDS) .  This  match-a- 
pattern- take -an- action  characteristic  can  permit  "smart"  behavior  on  the  part 
of  the  ISS,  and  if  the  ISS  is  truly  to  support  the  instructor  and  avoid 
inadvertent  actions,  very  complex  recognition  may  be  necessary.  The  actuation 
criteria  that  can  be  specified  for  a  given  ISS  was  therefore  a  fundamental 
determiner  of  ISS  performance.  It  should  be  noted  that  start-and-stop  condi¬ 
tions  are  often  difficult  to  describe  precisely  enough  for  computer  recog¬ 
nition.  Therefore  this  remains  one  of  the  primary  challenges  in  developing  a 
"smart"  ISS. 

Performance  Measurement.  Performance  measurement  is  an  ISF  of  an  ISS, 
and  in  fact,  such  systems  are  often  called  Performance  Measurement  Systems. 
Of  course,  performance  measurement  is  important  for  scoring  and  grading,  but 
is  also  important  as  an  adjunct  to  normally  available  simulator  variables  for 
the  purposes  of  control  of  ISS  instructor  support  features. 

Instructor  Support  Actions.  An  ISS  may  incorporate  a  number  of  ISFs 
which  require  direct  control  of  the  simulator  and  setup  conditions,  instructor 
console  displays,  insertion  and  removal  of  malfunctions,  display  of  diagnostic 
messages,  and  the  recording  of  detailed  data  for  post-simulator  use.  Appendix  F 
describes  the  comparison  of  instructor  support  actions  among  the  four  systems 
in  greater  detail. 

Instructor/ISS  Interaction.  Although  an  ISS  derives  much  of  its  effect¬ 
iveness  from  automatic  functions,  it  must  also  support  an  instructor  in  a 
flexible  manner,  allowing  the  instructor  to  override  and  re-direct  its  actions. 
For  example,  the  instructor  may  wish  to  vary  the  sequence  of  task  modules,  skip 
a  task  module,  or  alcer  the  conditions  under  which  a  malfunction  is  inserted. 
The  ability  for  flexible  interaction  between  instructor  and  ISS  was  examined 
during  comparisons  among  the  selected  systems  and  is  discussed  in  greater 
detail  in  Appendix  F. 

Conclusions 


A  view  has  been  taken  that  the  ISS  is  a  programmable  controller  and 
a  generator  of  information,  and  although  the  four  representative  ISSs  vary  in 
the  way  they  are  implemented,  all  fit  the  same  general  model.  The  method  of 
specifying  the  program  for  the  ISSs  varies,  but  the  task  module  concept  can  be 
used  for  initial  specification  for  any  ISS.  This  implies  a  data-driven 
system,  but  specification  in  this  manner  still  permits  a  large  degree  of 
design  freedom.  The  types  of  actuation  criteria  included  in  an  ISS  determined 
how  "smart"  the  ISS  can  be  in  behaving  "intelligently"  in  controlling  instuc- 
tional  events  and  features.  The  ISS  can  be  the  generator  of  a  large  amount  of 
different  types  of  information,  and  this  capability  depends  on  the  manner  in 
which  performance  measurement  is  implemented  and  the  types  of  data  recorded 
and  displayed.  Although  all  four  systems  provide  a  preprogrammed  automatic 
mode,  each  provides  some  manner  for  instructor  control  and  override,  allowing 
a  degree  of  flexibility  in  use  of  the  ISS  for  tailored  instruction.  All 


provide  instructional  support  through  the  control  of  performance  measurement, 
scoring,  displays,  malfunctions,  communications  and  data  recording.  These 
four  systems  provide  a  range  of  examples  that  characterize  the  current  tech¬ 
nology  and  provide  a  basis  for  the  generation  of  future  specifications. 

Hardware  and  Software  Implications 

The  purpose  of  this  section  is  to  present  findings  on  the  hardware  and 
software  implications  of  ISS  functional  requirements.  The  four  systems  were 
studied  from  the  systems  engineering,  hardware,  and  software  perspective,  and 
an  attempt  was  made  to  correlate  the  resulting  ISS  characteristics  with  the 
requirements  and  to  note  commonalities  among  the  four.  Cost  was  also  examined. 
However,  it  was  difficult  to  break  down  for  meaningful  comparison. 

All  four  systems  were  developed  as  adjuncts  to  existing  ATDs .  The 
information  collected  reflects  the  final  hardware  and  software  configurations 
used  for  the  AFTS®  F-4E  and  A-7D  operational  production  systems,  the  F-14  ISS 
research  and  development  system,  the  C-5A  PMS  research  and  development  system, 
and  the  B-52  ARPTT  operational  system. 

Hardware  and  Software  Components 

Table  3  contains  the  data  that  characterize  the  capability  of  the  hardware 
and  software  of  the  four  systems.  The  implications  of  each  characteristic  or 
associated  group  of  characteristics  are  described  further  below. 

Stations .  The  number  of  instructor  support  stations,  their  location,  and 
functions  provided  at  each  station  were  noted  as  being  functions  of  the 
existing  ATD  configuration  and  instructional  requirements.  For  example,  for 
the  B-52  ARPTT  ISS,  existing,  non-functional  displays  were  replaced  with  CRTs 
and  touch  pad  devices  to  allow  instructor  control  and  monitoring  from  either 
the  left  or  right  seat  on  board;  the  need  for  better  instructor  control  cc  the 
simulator  was  identified  as  a  priority  item  during  the  IOT&E  of  the  ARPTT 
prototype.  Concurrency  of  off-line  activity,  such  as  debriefing  using  off-line 
replay  with  real-time  monitoring,  is  also  a  consideration. 

Man-Machine  Interface.  The  selection  of  input  and  output  devices  which 
constitute  the  instructor  interface  with  the  ISS  is  driven  by  easily  used 
input  devices  and  easy  to  understand,  uncluttered  displays  within  the  con¬ 
straints  of  space  availability  and  the  state  of  current  technology.  Special 
function  keyboards,  coupled  with  functionally  grouped  menus  in  the  older 
systems,  led  to  touch-activated  devices  and  color  CRTs  in  the  B-52  ARPTT  ISS. 
In  the  AFTS®  and  F-14  ISS,  speech  generation  devices  provided  for  the  natural 
replay  and  provision  of  controller  advisories.  Speech  understanding  within  the 
AFTS®  facilitated  automation  of  the  controller  role  for  air-to-air  intercept 
training. 

Simulation  Interface.  In  two  of  the  fovir  systems  (AFTS®  and  C-5A)  a  data 
acquisition  unit  was  used  to  examine  data  that  flowed  between  the  simulation 
computer  and  the  simulated  cockpit  devices.  In  the  C-5A  PMS,  additional  data 
were  obtained  by  tying-in  between  the  IP  station  and  the  ATD's  input  processing 
computer.  In  the  F-14  ISS,  the  switch  unit  was  interposed  between  the  simu- 


lation  input/output  (I/O)  processor  and  the  simulation  computer's  shared 
memory.  The  data  acquisition  unit  utilized  on  the  AFTS®  and  C-5A  PMS  was 
designed  to  accommodate  2000  channels  and  burst  rates  of  at  least  750,000 
24-bit  words  per  second.  The  F-14  ISS  dealt  with  32-bit  words  sent  over  a 
2-Mbit  serial  stream.  The  B-52  ARPTT  ISS  was  unique  in  that  it  shared  memory 
with  the  simulation  computer.  All  interfaces  were  designed  to  capture  and 
send  information  on  ATD  parameters  necessary  to  monitor  and  control  the  ATD  on 
a  non-interference  basis.  Tradeoff  studies  were  performed  to  identify  the 
means  considered  most  cost-effective  for  the  existing  ATD  configuration. 

Computational  Svstem.  Ail  four  systems  were  implemented  as  stand-alone 
systems  that  utilized  their  own  processors  and  storage.  Design  tradeoff 
studies  performed  on  the  C-5  ATD  and  ARPTT  ATD  considered  the  possibility  of 
utilizing  existing  spare  capacity  within  the  ATD  host,  but  it  was  decided  that 
the  system  could  be  best  developed  by  using  additional  processors  to  minimize 
impacts  on  the  existing  ATD.  All  computers  were  commercially  available. 
Their  computing  capacity  is  expressed  in  terms  of  Data  General  Nova  instruction 
execution  and  FORTRAN  Whetstone  benchmarks  as  points  of  reference  in  Table  3. 
Three  of  the  four  systems  distributed  the  computing  task  further,  utilizing 
more  than  one  minicomputer. 

For  the  most  part,  the  processors  chosen  to  perform  the  ISS  control  were 
minicomputers.  When  the  first  prototype  AFTS®  was  developed,  one  of  the 
goals  was  to  show  that  the  system  could  be  built  using  minicomputer  technology, 
rather  than  mainframes.  Proven  effective  in  AFTS®,  similar  (but  of  the 
technology  commercially  available  at  the  time  of  their  procurement)  mini¬ 
computers  were  applied  to  the  F-14  ISS  and  the  C-5A  PMS. 

The  Perkin- Elmer  32/D  was  a  departure  from  the  above  in  that  the  ISS 
was  procured  as  part  of  an  upgrade  of  the  entire  ATD  to  meet  a  15 -year  life 
cycle.  The  32/D  was  a  cost-effective,  vendor- supported  upgrade  for  the 
existing  ATD;  the  selection  of  an  additional  32/D  and  a  shared  memory  interface 
was  a  good  logistical  choice  for  the  ARPTT  ISS,  as  shown  in  the  life  cycle  cost 
study  performed  as  part  of  the  ARPTT  upgrade  study. 

System  Performance.  System  performance  takes  into  account  whether  the 
capacity  of  the  computational  system  adequately  supported  the  functional 
demands  upon  the  system,  such  as  monitoring  cycle  time,  number  of  simulation 
variables,  expected  task  module  concurrency,  and  the  software  architecture. 

In  all  four  systems  graphics  computation  was  to  some  extent  off-loaded 
onto  the  graphics  device.  In  the  F-14  ISS  and  C-5A  PMS  systems,  a  dedicated 
minicomputer  was  allocated  for  graphics  processing.  It  should  be  noted  that 
the  processor  used  for  graphics  on  the  C-5A  PMS  proved  to  be  inadequate  due  to 
demands  on  its  capability  to  load  files  off  disk. 

The  F-14  ISS  computer  performed  adequately  for  a  single,  off-line  or 
real-time  activity;  the  disk  file  management  demands  of  the  software  archi¬ 
tecture  precluded  effective  concurrent  activity.  The  C-5A  PMS  computer 
effectively  handled  all  activity  except  for  the  monitoring  of  momentary 
switches.  An  attempt  at  increasing  the  cycle  time  to  200  milliseconds  from 
800  milliseconds  was  thwarted  by  the  lack  of  sufficient  spare  computing  power. 
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The  higher  performance  capacity  of  the  B-52  ARPTT  ISS  32/D  allowed  all  ISS 
control  and  graphics  to  be  hosted  in  one  computer.  AFTS®  and  ARPTT  perform 
adequately  given  the  required  functionality,  but  with  little  spare  computing 
and  memory  capacity. 

Software  Architecture.  In  all  four  systems,  an  attempt  was  made  to  allow 
for  changes  in  training  tasks  to  be  accomplished  without  requiring  reprogramming 
of  system  software.  This  was  accomplished  via  use  of  disk-based  data  files 
which  were  predefinable  off-line  using  editors  and  preprocessors  and,  in  all 
systems  except  the  AFTS®,  via  runtime  control  actions.  It  is  of  interest  to 
note  that  more  than  just  parameter  editing  is  necessary  to  accommodate  changes 
expected  in  procedural  tasks,  i.e.,  checklists,  as  shown  especially  in  C-5A 
PMS  in  which  a  language  was  developed. 

As  discussed  in  the  previous  section,  task  modules  were  embodied  in 
code  in  the  AFTS®  and  in  data  files  in  the  other  three  systems.  The  latter 
three  systems  were  data-driven  in  the  sense  that  the  task  module  definition 
provided  data  which  described  events  that  initiated  activity  on  the  part  of 
the  ISS.  In  the  C-5A  PMS,  the  activity  was  hardcoded.  In  the  B-52  ARPTT  and 
F-14  ISSs,  some  of  the  parameters,  e.g.,  diagnostic  feedback  messages  or 
measurement  algorithm,  were  identified  by  data  within  the  task  module  defi¬ 
nition. 

The  C-5A  PMS  and  AFTS®  used  fixed  (synchronous)  execution  cycles  with 
code  being  scheduled  and  executed  on  one  of  two  available  cycles,  200  or  800 
milliseconds.  The  F-14  ISS  and  B-52  ARPTT  ISS  had  synchronous  and  asynchronous 
(on  request)  components.  The  synchronous  components  took  care  of  graphic 
updates,  event  detection  (processing  of  simulator  variables  of  interest  to  see 
if  an  action  was  required),  and,  in  the  case  of  the  ARPTT  ISS,  the  processing 
of  continuous  flight  task  requirements .  The  asynchronous  components  performed 
activities  as  required,  when  required.  Thus,  instructor  control  requests  or 
student  actions  could  be  acted  upon  when  the  trigger  (start,  stop,  procedural 
step,  out-of- tolerance ,  etc.)  event  was  sensed.  Activity  priorities  could  be 
adjusted  to  match  user  requirements. 

Software  Components.  The  procurement  of  an  ISS  can  address  needs  and 
budgetary  tradeoffs  for  a  minimal  to  an  all-encompassing  ISS.  Each  ISS 
configuration  can  address  a  different  set  of  major  functions.  Major  functional 
areas  identified  through  the  study  of  training  requirements  include:  display 
and  control,  measurement,  brief/debrief,  tutorial,  and  training  management. 
These  five  areas  map  to  ISFs  in  the  following  manner: 

1.  Display  and  control  --  IOS  Display  Control  and  Formatting,  Initial 
Conditions,  Real-time  Simulation  Variables  Control,  Malfunction  Control, 
Freeze 


2.  Measurement  --  Automated  Performance  Measurement,  Procedures  Moni¬ 
toring,  Scenario  Control 

3.  Brief/Debrief  --  Briefing  utilities,  Hardcopy/Printout,  Remote 
Graphics  Replay 


4. 


Tutorial 


Tutorial 


5.  Training  Management  --  Data  Storage  and  Analysis 

At  a  minimum,  an  ISS  requires  display  and  control  functions  that  allow 
the  control  of  the  ATD  and  the  real-time  monitoring  of  the  student.  Given 
this  core  capability  measurement,  then  brief/debrief,  tutorial,  and  training 
management  can  be  added  as  required  to  procure  a  system  that  meets  the  needs  of 
the  user  community.  Table  3  presents  Information  on  software  components  Imple¬ 
mented  for  the  four  systems,  broken  down  by  functional  area  to  give  a  frame  of 
reference  for  future  procurements. 

The  software  for  each  of  the  systems  was  written  using  combinations 
of  assembly  language  and  available  high  level  languages,  such  as  FORTRAN  and 
Data  General  Corporation's  ALGOL  (DG/L).  Note  that  source  lines  of  code 
(SLOC)  numbers  identified  in  Table  3  include  extensive  assembly  language 
lines,  resulting  in  much  higher  numbers  for  the  AFTS®.  Operating  systems  and 
utilities  commercially  available  for  the  hardware  were  adopted  for  both  the 
development  and  runtime  operation  of  the  systems.  All  operating  systems  were 
multitasking,  able  to  keep  track  of  more  than  one  system  activity  at  a  given 
time.  Also,  the  F-14  ISS  took  advantage  of  its  multiprogramming  operating 
system's  messaging  capability  to  implement  asynchronous  system  control. 

Common  Characteristics 

Given  the  collected  data  just  discussed,  major  characteristics  common 
among  the  four  ISS  systems  lead  to  the  definition  of  a  stand-alone  system  for 
ISSs  which  can  be  added  to  existing  ATDs.  This  stand-alone,  modular  ISS  has 
the  following  features: 

1.  All  ISS  processing  is  isolated  from  the  simulation  processing. 

2.  Data  are  passively  shared  with  the  simulator. 

3.  The  ISS  can  be  added  with  minimal  modification  to  an  existing  ATD. 

4.  The  system  does  not  require  large,  i.e.,  mainframe,  computers. 

5.  The  system  attempts  to  provide  user  stations  that  are  tailored  for 
effective  instructor  use. 

6.  The  training  tasks  are  specifiable  and  updatable  without  requiring 
system  reprogramming. 

7.  Software  is  functionally  modularized. 

8.  Commercially  available  software  and  hardware  are  utilized  to  the 
greatest  extent  possible. 

Hardware  and  its  accompanying  software  development  environment  can  be  spec- 
|  ifically  selected  to  support  the  ISS  functions  or  can  be  expanded  in  function¬ 

ality  without  impacting  the  simulation.  Additionally,  adverse  impacts  on 


the  existing  ATD  utilization  by  ISS  development  can  be  minimized.  For  example, 
the  AFTS®  was  added  to  existing  F-4E  and  A- 7  ATDs  in  about  a  week.  The  AFTS®, 
F-14  ISS,  and  C-5A  PMS  could  also  be  switched  completely  off  as  necessary  to 
allow  use  of  only  the  pre-existing  ATD. 

The  synchronous  rate  required  for  simulation  processing  is  not  required  for 
ISS  processing.  For  example,  the  F-14  ISS  was  attached  to  the  2F95  ATD  which 
had  a  cycle  time  of  20  Hz.  The  F-14  ISS  successfully  performed  all  display 
and  monitoring  functions  using  a  5 -Hz  simulation  variable  monitoring  cycle 
coupled  with  asynchronous  processing  of  other  activity.  The  B-52  ARPTT  ISS  was 
designed  in  a  similar  fashion,  allowing  for  monitoring  of  events  of  interest, 
processing  of  continuous  flight  variables,  and  updating  the  displays  at  a  10- 
Hz  rate.  The  application  of  a  small,  synchronously  executing  kernel  of 
software  dedicated  to  the  detection  of  events  of  interest  could  possibly  have 
alleviated  the  problem  of  detecting  momentary  switches  in  C-5A  PMS. 

Software  modularity  allows  for  addition  of  functionality  in  phases.  For 
example,  a  tutorial  capability  was  added  to  the  F-14  ISS  after  all  other 
functionality  was  completed.  The  B-52  ARPTT  ISS  was  delivered  in  two  phases: 
the  first  phase  provided  monitoring  and  display  functions,  while  the  second 
provided  measurement  and  training  management  functions.  The  ISS  was  operational 
after  delivery  of  the  first  phase  and  down  time  was  minimal  for  the  installation 
of  the  second  phase. 

The  above  constitutes  a  baseline  description  of  what  fairly  high  level 
commonalities  may  be  carried  forth  into  a  generic  architecture  for  an  ISS.  The 
commonalities  break  down,  when  going  too  far  past  the  functional  requirements 
of  these  systems,  to  a  lower  level  of  implementation  detail. 

Note  that  the  systems  represent  a  progression  from  AFTS®,  which  was  archi¬ 
tected  in  1973  (based  on  studies  performed  since  1969),  to  C-5A  PMS  and  F-14 
ISSa,  which  were  developed  in  1978,  to  the  B-52  ARPTT  ISS,  which  was  designed  in 
1981  and  1982.  The  C-5A  PMS  did  indeed  reuse  pieces  of  the  AFTS®  executive 
control  software,  and  the  B-52  ARPTT  ISS  transferred  the  F-14  ISS  software 
architectural  concepts  to  a  different  training  problem  and  hardware  suite. 
These  transfers  were  successfully  accomplished  but  not  without  a  great  deal  of 
additional  customization  and  development.  This  again  is  evident  in  the 
funding  required  to  build  the  production  AFTS®  based  on  work  done  on  the 
prototype  AFTS®  (  see  the  following  discussion  on  cost  factors) . 

The  development  of  these  systems  has  capitalized  on  previous  lessons 
learned  and  advancements  in  technology,  as  well  as  focusing  on  the  unique 
requirements  of  each  of  the  procurements. 

Cost  Factors 


Historical  costs  could  not  be  accurately  separated  into  the  same  cost 
elements  across  the  four  systems,  since  the  work  breakdown  structure  was 
different  in  every  case.  The  significant  results  were  in  the  area  of  relative 
contractor  development  and  procurement  costs  and  identification  of  cost 
drivers,  rather  than  the  absolute  value,  since  only  contractor  costs  were 
reviewed. 


The  relative  costs  for  the  AFTS®  production  and  C-5A  PMS  are  shown  in 
Table  4  as  points  of  reference.  Note  that  the  C-5A  PMS  and  AFTS®  costs  do 
not  reflect  the  costs  associated  with  the  research  and  development  studies 
which  preceded. 


Table  4.  Relative  Development  and  Production  Costs 


Cost  Category 

Svstems 

AFTS® 

A-7D:  1  proto,  4  production 
F-4E:  1  proto,  15  production 

C-5A  PMS 

1  research/develop¬ 
ment  system 

Management 

7  % 

9  % 

Software  Engineering 

8  % 

47  % 

Hardware  Engineering 

6  % 

32  % 

(engineering  and 

Manufacturing/ 

39  % 

procurement) 

Procurement 

per  unit 

2  % 

Installation 

5  % 

4  % 

Provisioning 

18  % 

N/A 

Data 

13  % 

7  % 

Quality  Assurance 

2  % 

N/A 

Final  Contract  $ 

9,690,928 

1,237,252 

Reference  Year 

1978 

1980 

Adjusted  Amount  in  1984  $ 


17,637,489 


1,670,290 
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Software  costs  are  identifiable  as  cost  drivers  and  have  traditionally  been 
difficult  to  estimate.  Table  5  shows  the  relative  costs  of  the  different 
software  modules  implemented  in  C-5A  PMS  and  F-14  1SS.  Estimated  source  lines 
of  code  figures  are  given  to  provide  a  frame  of  reference  for  the  magnitude  of 
the  software  development  effort.  Note  that  the  referenced  numbers  contain  a 
mix  of  assembler  (both  ISSs),  FORTRAN  (C-5A  PMS),  and  DG/L  (both  ISSs) . 
Comparable  costs  are  not  available  for  the  other  programs . 


Table  5.  Relative  Software  Costs 


Life  cycle  costs  were  addressed  in  the  B-52  ARPTT  ISS.  The  elimination 
of  a  console  operator  by  adding  the  ISS,  and  provision  of  computers  supportable 
for  a  15-year  life  cycle  were  major  factors.  Again,  these  costs  reflect  the 
objective  of  the  ARPTT  upgrade  study,  to  upgrade  an  existing  prototype  ATD  for 
a  15 -year  life  cycle. 


As  specified  in  the  contracted  Statement  of  Work,  selection  of  the 
Guidelines  format  was  to  be  made  by  the  Air  Force.  Suggested  criteria  for 
selection  of  a  format  were  that  it  provide  condensed  yet  comprehensive  informa¬ 
tion  and  that  it  be  easy  to  use  and  reference. 

The  Guidelines  are  intended  to  educate  the  reader  about  the  instructor 
support  capabilities,  as  well  as  to  supplement  other  current  specification 
guides.  For  this  reason,  the  narrative  format  style  was  recommended  because 
it  can  present  a  deeper  layer  of  information.  This  format  was  subsequently 
selected.  The  IOS  Working  Group  members  selected  the  Information  Mapping® 
layout  for  the  visual  presentation  of  the  Guidelines. 


.vrv.v.v .V/V.  -V  : 
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It  was  felt  that  a  single  guidelines  document  to  be  used  by  the  spectrum 
of  individuals  involved  in  specifying  ATDs ,  rather  than  several  volumes  geared 
to  separate  audiences,  would  promote  a  greater  degree  of  mutual  understanding 
of  ATD  requirement  specifications.  To  facilitate  information  access,  tabs  and 
an  index  have  been  included. 

Computer -readable  diskettes  containing  the  Guidelines  were  provided  to 
SimSPO  upon  submittal  of  the  final  deliverable  of  the  ISF  Guidelines.  With 
regular  on-line  update,  the  Guidelines  can  serve  as  a  repository  for  lessons 
learned  to  benefit  all  users. 


The  ISF  Guidelines  were  developed  as  a  result  of  all  of  the  efforts  des¬ 
cribed.  The  review  of  ATD  acquisition/specification  process  and  instructor 
support  feature  analysis  provided  valuable  insight  into  user  requirements. 
The  internal  ISS  comparison  and  the  examination  of  hardware/software  impli¬ 
cations  provided  important  lessons  learned  about  the  development  and  utilization 
of  state-of-the-art  systems.  It  is  hoped  that  use  of  these  Guidelines, 
written  and  formatted  with  the  readers  in  mind,  will  lead  to  better  written 
specifications  for  ISS  capabilities  of  future  aircrew  training  devices. 


IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


Front-end  analysis  for  both  student  and  instructor  requirements  Is  needed 

To  ensure  instructional  system  performance  fully  supportive  of  the  specific 
requirements,  a  comprehensive  front-end  analysis  of  instructor  requirements,  as 
well  as  student  training  tasks,  must  be  completed.  Requirements  for 
instructional  systems  which  are  copied  from  seemingly  similar  recent  procure¬ 
ments  may  at  best  be  approximations  of  what  is  actually  required.  Efforts 
committed  to  proper  planning,  analysis,  and  specification  of  the  instructional 
system  will  prevent  specifications  that  do  not  meet  user  needs. 

Instructor  training  is  needed 

There  have  been  several  documented  studies,  including  the  one  conducted 
during  Phase  II  of  this  project,  which  point  out  that  instructors  are  not 
properly  trained  in  the  use  of  ATDs,  especially  in  the  area  of  instructional 
features.  A  well  designed  instructional  system  for  the  ATD  greatly  improves 
the  potential  for  simulator  training  in  many  ways  that  are  not  available  when 
using  the  actual  aircraft.  This  potential  will  not  be  realized  (and  the 
features  will  be  ignored)  if  minimal  time  has  been  allocated  for  training  the 
instructor  in  the  use  of  the  instructor  support  features,  or  if  the  instructor 
infrequently  uses  the  simulator. 

The  ISF  Guidelines  have  been  carefully  written  in  user-specific  terms  to 
provide  operational  definitions  and  lessons  learned.  These  guidelines  would 
be  a  valuable  tool  during  initial  formal  training  to  introduce  the  instructor 
to  instructional  features  and  their  intended  use. 


Ongoing  instructor  effectiveness  would  also  benefit  from  automated 
instructor  support  system  capabilities  specific  to  a  particular  ATD  if  they 
were  better  understood  with  respect  to  specific  training  objectives.  Therefore, 
the  design  should  include  provisions  for  built-in  instructor  tutorials  and  help 
features  specific  to  the  device. 

Task  modules  provide  operational  data  required  for  the  instructional  system 
to  support  user  needs 

In  order  to  provide  user- oriented  and  meaningful  "on-line"  support,  an 
instructional  system  must  "know"  where  the  student  is  in  a  training  mission 
and  to  take  preprogrammed  action  under  appropriate  circumstances.  The  task 
module  (user-oriented  building  blocks  which  have  been  transformed  into  software 
data  files)  has  evolved  as  a  central  concept  in  this  regard,  containing 
the  logic,  performance  algorithms  and  criteria,  displays,  data  recording 
and  other  actions  to  be  taken.  The  task  module  provides  a  means  for  commun¬ 
ication  between  the  operational  personnel  and  the  contractor:  it  provides 
the  means  for  operational  personnel  to  specify  precisely  to  what  extent  the 
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ISS  is  going  to  support  the  achievement  of  each  instructional  objective;  and  it 
provides  the  system  developers  with  the  means  for  controlling  the  system  to 
achieve  efficient  and  effective  training. 


It  is  important  for  preliminary  design  of  an  instructional  system  that  the 
training  objectives  of  a  device  be  clearly  defined  early  on.  These  objectives 
may  be  directly  correlated  with  a  list  of  task  modules  (as  described  above) 
which  may  then  be  used  as  part  of  the  design  criteria. 

The  instructional  system  should  be  kept  current 

An  ISS  uses  a  great  deal  of  information  specific  to  the  training  to  be 
administered.  This  information  includes  definitions  for  each  mission  profile 
in  the  syllabus,  information  for  each  flight  maneuver  and  procedure,  together 
with  performance  measurement  criteria,  and  definitions  of  each  computer¬ 
generated  display  (including  navigation,  approach,  and  departure  templates). 
These  data  must  be  kept  current  with  changes  in  training  requirements  and 
flight  procedures;  otherwise,  the  instructional  system  becomes  unusable  or 
ineffective  in  the  areas  of  change.  This  is  one  of  the  important  lessons 
learned  from  the  systems  that  have  been  tested  in  operational  environments. 

Consequently,  instructional  system  design  should  include  provision  for 
economic  and  efficient  update  and  revision.  This  has  been  successfully  accom¬ 
plished  using  the  task  module  concept  (database -driven  architecture) .  The 
dynamically  changing  data  described  above  are  transcribed  into  computer  files 
and  contained  in  a  database.  The  files  are  then  used  by  a  database -driven 
software  while  the  system  is  in  operation. 

Transcribing  operational  data  into  the  database  files  would  best  be 
accomplished  by  a  software  utility,  thus  preventing  dependence  on  system 
specific  software  skills  and  expertise  that  is  both  costly  and  time-consuming. 
This  update  could  then  be  accomplished  at  the  field  activity  in  parallel  with 
any  operational  objective/procedure  change. 

Levels  of  automation  should  depend  on  the  training  objective 

The  state  of  the  art  in  instructional  systems  is  fully  capable  of  providing 
complete  automation  of  a  training  scenario,  including  the  preprogramming  of  all 
required  inputs  during  a  session.  This  type  of  ATD  control  may  be  very 
effective  when  the  instructor's  presence  is  primarily  to  monitor  (e.g., 
standardization/evaluation  events  or  total  mission-oriented  events).  This 
type  of  automation  and  control  enhances  standardization  for  evaluation  purposes 
and  relieves  the  instructor  of  non- instructional  tasks  during  mission -type 
training. 

However,  total  automation  may  be  totally  undesirable  for  training  where  the 
instructor  should  be  actively  involved  in  the  training  process  (e.g.,  UPT 
training  where  motor  skills  are  learned  and  reinforced).  In  these  cases, 
total  automation  restricts  the  instructor's  capability  to  tailor  the  session  to 
the  student's  needs. 


The  design  of  the  instructional  system  should  be  especially  sensitive  to 
the  automated  control  features  and  provide  the  levels  required  by  the  training 
objectives  specific  to  that  device.  Providing  varying  levels  of  automation 
that  can  be  selected  by  the  instructor  as  part  of  the  exercise  definition 
process  provides  the  flexibility  to  cover  a  broad  spectrum  of  training 
objectives  of  many  ATDs . 

Automated  performance  measurement  should  be  designed  as  an  aid  to  the  instructor 

In  the  past,  performance  measurement  was  too  inflexible  and  rudimentary  to 
provide  meaningful  feedback.  For  example,  some  performance  measurement 
systems  provide  single  aircraft  parameters  that  have  no  meaning  when  presented 
in  isolation.  Such  parameters  should  be  presented  in  combination  with  other 
parameters  and  student  actions  to  present  a  more  complete,  and  therefore 
more  accurate,  evaluation  of  the  student's  performance  on  an  objective.  In 
other  cases,  performance  measurement  systems  have  provided  a  single  score  for 
a  maneuver  without  any  supporting  data.  This  type  of  measurement  is  also 
usually  ignored  by  instructors  and  students  because  it  provides  no  information 
with  respect  to  how  the  score  was  obtained. 

State-of-the-art,  automated,  performance  measurement  technology  evaluates 
performance  by  objective,  taking  into  account  all  of  the  actions  required  to 
perform  the  maneuver.  A  review  of  the  measurement  of  each  action,  along  with 
the  evaluation  criteria,  should  be  made  available  such  that  the  student  and  the 
instructor  can  determine  both  whether  a  criterion  was  met  and  how  it  was 
performed. 

Automated  performance  measurement  should  be  designed  to  aid,  not  replace, 
the  instructor  in  his  evaluation  of  student  performance. 

Requirements  should  not  be  overspecified 

The  specification  of  instructional  features  should  be  based  on  functio¬ 
nality  rather  than  technology.  Within  the  basic  instructional  system  archi¬ 
tecture,  use  of  available  technology  is  to  be  encouraged  by  not  over-specifying 
hardware  or  software  components.  Technology  changes  daily:  computer  hardware 
costs  are  decreasing  as  performance  continually  increases;  new  input  and  display 
devices  are  continually  being  introduced;  software  technology  is  on  the  verge 
of  making  major  "’dvances  with  Ada  as  a  programming  language;  and  artificial 
intelligence  and  other  approaches  are  emerging.  Therefore,  specification  of 
functionality  and  performance  from  a  user's  perspective  is  imperative.  This 
will  allow  contractor  latitude  in  providing  SimSPO  with  a  spectrum  of 
alternatives  that  will  maximize  the  application  of  current  technological 
advances  and  current  standards. 

The  should  be  maintained  as  a  dynamic  document 

The  product  of  this  project,  the  ISF  Guidelines,  is  intended  to  effectively 
transition  lessons  learned  and  proven  technology  from  advanced  instructional 
systems  into  the  operational  training  environment.  As  rapid  advances  in 
weapon  systems  technology  occur,  progress  in  instructional  systems  technology 
is  expected.  With  the  introduction  of  software-driven  cockpits,  tactics  and 
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procedures  required  to  overcome  new  threats  and  adversaries  cannot  be  pre¬ 
dicted.  Instructional  systems  technology  will  meet  this  challenge  with 
innovations  in  hardware,  software,  and  yet  undiscovered  technologies.  The 
guidelines,  too,  should  accommodate  these  changes  with  continued  update  and 
revision. 
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APPENDIX  A 

TRAINING  SITES  VISITED 


Training 

Device _ Location _ Date 


AFTS  A-7D 

Davis -Monthan  AFB 

12/7/84 

A- 10 

Davis -Monthan  AFB 

12/7/84,  1/30/85 

ARPTT  ISS 

Castle  AFB 

1/15/85 

B-52 

Castle  AFB 

1/15/85 

C-5A  PMS 

Altus  AFB 

12/12/84 

C-130 

Little  Rock  AFB 

12/11/84 

F-15 

Luke  AFB 

11/15/84,  1/29/85 

F-16 

Luke  AFB 

11/15/84,  1/29/85 

T- 37  (T-50) 

Williams  AFB 

11/13/84 

11/13/84 


T-38  (T-51) 


Williams  AFB 


APPENDIX  B 


INTERVIEW  GUIDE 


This  interview  guide  was  used  merely  to  guide  the  interview  to  bring  out 
the  issues  and  not  to  collect  data.  Because  the  number  of  instructor  support 
features  incorporated  into  each  system  varied  significantly  from  one  ATD  to 
the  next,  this  form  was  used  as  a  general  guide.  The  resulting  data  has  been 
summarized  in  Section  III,  Results,  Instructor  Support  Feature  Analysis. 
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INTERVIEW  GUIDE 
SYSTEM  LEVEL  CONSIDERATIONS 


Contact:  _  Phone:  £ _ 1 

Position:  _ 

Site:  _  ATD:  _ 


1.  What  are  the  training  objectives? 

2.  What  are  the  specific  tasks? 

3.  What  is  the  performance  criteria? 

4.  Can  you  provide  any  grade  sheets  or  records  of  accomplishment? 

5.  Does  the  simulator  capabilities  match  the  stated  requirements? 

a.  Does  it  train  to  the  objectives?  _  yes  _  no 

b.  If  not,  what  the  are  shortfalls?  Is  it  because  the 
can't  do  it,  or  is  it  because  it's  not  used? 

6.  Are  there  advanced  instructor  support  features? 

If  yes,  do  they  accomplish  the  intended  purpose? 

If  not,  why  not? 

Yes  No 

Scenario  Generation  _  _ 

Preprogrammed  Missions  _  _ 

ISS  CONTROL  FUNCTIONS 

Simulation  Variables  Control _  _ 

Malfunction  Control  _  _ 

Repositioning  _  _ 

Display  Control  _  _ 


simulator 
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Yes  No 


> 


INSTRUCTOR  CONTROL  OPTIONS 

Display  _  __ 

Scenario  Changes  _  _ 

Mode  Changes  _ 

Tutorial  _  _ _ 

Algorithms/Assessments  _ 

Grading  Criteria  ____  _ 

Data  Storage  &  Analysis  _ _  ___ 

Brief  _  _ 

Debrief  _ _ 

7 .  Does  the  course  syllabus  match  what  is  actually  taught? 
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INTERVIEW  GUIDE 


TASK  LEVEL  CONSIDERATIONS 

ATTEMPT  TO  AUGMENT  OUR  COLLECTION  OF  REPORTS 

Task  listing  for  Simulator  Training 

Material  for  the  training  of  instructors,  Instructor  Guides 
Stan/Eval  documents 
Student  guides,  handouts 

OBSERVE  SOME  TRAINING  IN  EACH  SIMULATOR 

PERFORMANCE  MEASUREMENT  DISCUSSIONS 

In  the  context  of  the  following  matrix  (as  appropriate  to  the  training) 


!  !  FLIGHT  !  PROCEDURES  ! 


!  !  !  ! 

!  NORMAL  !  !  ! 

!  !  !  ! 


!  !  !  ! 

!  EMERGENCY  !  !  ! 

!  !  !  ! 


!  !  !  ! 

!  TACTICAL  !  !  ! 

!  !  !  ! 


Walkthrough  a  selected  task  in  each  category 

1.  Looking  at  fine-grained  performance?  End  results? 

2.  Common  errors? 

3.  Standards  of  performance? 

4.  Scoring/grading  criteria?  Forms? 

5.  What  is  included  in  the  debrief  of  the  task? 

6.  Accessibility  of  Performance  Measurement  information 


Listen  for: 


Instructor  workload  level 
Information  not  detectable  by  machine 
Difficult-to-define  or  to-evaluate  task  factors 
Uncertain,  complex  task  sequencing 

Out-of-the-ordinary  performance  measurement  algorithms 


COLLECT  CONTACTS  FOR  FOLLOWUP  IF  NEEDED 


APPENDIX  C 


COURSE  DOCUMENTATION 

A- 10 

IOS  Manual,  Upgrade  Training  Course  (not  dated) 
Flight  Objectives  Pamphlet  (8/84) 

TAC  Syllabus  (8/84) 

Gradesheet 

B-52 


Training  Program  WST  Coursebook  (not  dated) 

Console  Familiarization  Course  (1984) 

WST  OIS  Console  Operations  Guide  Vol.  I  and  II  (8/84) 
WST  DDS  Console  Operations  Guide  (8/84) 

Test  Option  5  Scenario  Description  (not  dated) 


C- 130 


Pilot  Study  Guide  Part  I,  Pilot  Initial  Qualification  Course  (10/82) 
Student  Study  Guide  Part  II,  Tactical  Mission  Qualification  Training 
(12/82) 

Instructor  Guide  Part  II,  Pilot  Requalification/Upgrade  Course  (1/83) 
Instructor  Guide,  Navigator  Mission  Qualification  (12/82) 

Mission  Profiles  I  -  V  (not  dated) 

Nullmeyer ,  R.T.,  and  Rockway,  M.R.  Effectiveness  of  Weapon  System 

Trainers  for  Tactical  Aircrew  Training.  In  Proceedings  of  Inter¬ 
service/Industry  Training  Equipment  Conference  and  Exhibition, 
October  1984. 

Partial  Preliminary  Simulator  Instructor  Guide  for  Tactical  Mission 
Qualification  Training  (12/82) 

Flight  Simulator  Operating  Instructions  (10/82) 


C- 141 


SIM/CPT  Instructor  Guide,  Pilot  Initial  Qualification  Course  (11/82) 
Flight  Instructor  Guide,  Pilot  Initial  Qualification  Course  (1/81) 
Instructor  Guide,  Pilot  Airdrop  Qualification  Course  (11/83) 

Instructor  Malfunction  Guide,  Flight  Engineer  Initial  Qualification 
Course  (3/84) 

Flight  Instructor  Guide,  Navigator  Airdrop  Mission  Qualification  Course 
(3/83) 

Task  and  Objectives  Document,  Loadmaster  Airdrop  Qualification  Course 
(10/81) 
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Operational  Training  Course  (10/81) 

Simulator  Instructor  Pilot  Upgrade  Procedures  (7/83) 

Instructor  Operator  Guide,  F-15  Simulator  (7/83) 

F-16 

Sasic  Operational  Training  Course  (1/84) 

Wordstar  Lesson  Plans  (1984) 

Gradesheets 

Instructor  Handbook  (3/82) 

KC-135 

Pilot  WST  Coursebook  (1/84) 

Navigator  WST  Coursebook  (1/84) 

T-37 

Instrument  Program  (3/83  and  9/84) 

Syllabus  of  Instruction  for  Undergraduate  Pilot  Training  (T-37/T-38)  (8/83) 
T-50  IFS  Mission  Guide  (3/83) 

T-  38 

Syllabus  of  Instruction  for  Undergraduate  Pilot  Training  (T-37/T-38)  (8/83) 


APPENDIX  D 

SYSTEMS  WITH  ADVANCED  INSTRUCTOR  SUPPORT  CAPABILITIES 

Automated  Flight  Training  System  (AFTS®).  Developed  in  the  late  1970 's, 
the  Automated  Flight  Training System (AFTS®)  was  installed  on  the  Air  Force 
F-4E  and  A-7D  flight  simulators  for  use  by  the  Tactical  Air  Command  (TAC)  and 
the  Air  National  Guard.  This  system  provides  the  capability  to  run  preplanned 
automated  training  sequences  without  interfering  with  existing  trainer  perfor¬ 
mance  and  allows  a  set  of  training  courses  to  be  run  through  with  minimal 
instructor  intervention.  The  AFTS®  scores  the  aircrew  on  task  performance, 
and  on  satisfactory  score  achievement;  the  system  automatically  advances  the 
student  to  more  difficult  tasks.  This  automated  adaptive  training  is  provided 
for  the  following  exercises:  instrument  maneuvers,  instrument  penetration/ 
approaches,  instrument  departures,  radar  navigation,  normal  and  emergency 
procedures,  ground  attack  radar,  air-to-air  intercept,  and  weapon  scoring. 

F-14  Instructor  Support  System  (ISS).  The  F-14  ISS  was  an  R&D  prototype 
developed  in  1981  which  "strapped  on"  to  the  F-14  Operational  Flight  Trainer 
at  Miramar  Naval  Air  Station.  This  system  was  designed  to  provide  state-of- 
the-art  instructor  support  functions  in  the  areas  of  procedural  monitoring, 
performance  measurement,  briefing,  debriefing,  graphics  replay,  record  keeping, 
and  instructor  training  via  a  built-in  tutorial.  In  addition,  this  system 
provides  instructor-oriented  simulation  control  according  to  the  training 
objective.  The  ISS  was  developed  and  tested  on  site  in  the  operational 
environment  to  provide  direct  user  feedback. 

C-5A  Performance  Measurement  System  (PMS).  Developed  in  1982,  the  C-5A 
PMS  was  an  R&D  prototype  developed  to  "strap  on"  the  C-5A  flight  simulator.  It 
provided  such  additional  training  capabilities  as  preprogrammed  mission 
scenario  design,  real-time  aircrew  performance  measurement  and  instructor 
feedback,  and  post-mission  data  retrieval  and  analysis.  Various  levels  of 
statistical  performance  data  were  generated  and  recorded  by  the  C-5A  PMS 
during  a  mission.  These  data  could  then  be  retrieved  by  research  personnel  at 
any  time  for  the  purpose  of  performing  statistical  analysis. 

Aerial  Refueling  Part  Task  Trainer  Instructor  Support  System  (ARPTT 
ISS) .  The  Air  Refueling  Part  Task  Trainer  Instructor  Support  System  (ARPTT 
ISS),  Installed  on  the  ARPTT  simulator  at  Castle  AFB  in  1984,  is  an  instructor 
support  system  that  allows  the  instructor  to  operate  the  simulator  with  greater 
ease  and  more  flexibility.  Much  of  the  technology  used  in  the  F-14  ISS  design 
was  applied  in  the  ARPTT,  including  performance  measurement,  procedures 
monitoring,  and  record  keeping.  An  additional  feature  provided  curriculum 
managers  with  trainer  utilization  and  training  effectiveness  data. 
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APPENDIX  E 


SYSTEMS  DOCUMENTATION 


AFTS® 

Training  Specification  for  AFTS  for  A-7D  (6/77) 

Training  Specification  for  AFTS  for  F-4E  (7/78) 

Performance  Specification  for  AFTS  for  A-7D  (6/78) 

Performance  Specification  for  AFTS  for  F-4E  (6/78) 

Software  Users  Guide  for  AFTS  for  F-4E  (4/79) 

Program  Source  Listings 

Grunzke ,  P .  Evaluation  of  the  Automated  Adaptive  Flight  Training  System's 
Air-to-Air  Intercept  Performance  Measurement.  AFHRL-TR-78-23 .  Air 
Force  Human  Resources  Laboratory,  Williams  Air  Force  Base,  AZ.  Julv 
1978. 

AFT  Program  Description,  5/72 
Automated  Weapon  System  Trainer,  6/70 

B- 52  ARPTT  ISS 

Study  on  the  Refurbishment  of  Aerial  Refueling  Part  Task  Trainer  (ARPTT) 
to  Extend  its  Life  Expectancy  -  Technical  Report  (10/30/81) 
Functional  Design  Document  -  ARPTT  ISS  (11/30/84) 

Instructor  Guide,  B-52  Training  Program  Pilot  BPAT  (not  dated) 

ARPTT  Training  Program  5/84 
Program  Source  Listings 

C-5A  PMS 

C-5  Course  Summary  Document,  Pilot  Initial  Qualification  Course  (1/82) 
CPT/SIM/FLT  Student  Guide,  Pilot  Initial  Qualification  Course  (2/81) 
CPT/SIM/FLT  Instructor  Guide,  Pilot  Initial  Qualification  Course  (1/83) 
C-5  Pilot  Master  Task  Listing  (3/83) 

Operations  Manual  -  PMS  for  the  C-5A  Simulator  (9/83) 

System  Specification  (Parts  I,  II  and  III)  (12/82) 

Program  Source  Listings 

Swink,  T.R. ,  Butler,  E.A.,  Lankford,  H.E.,  Miller,  R.M. ,  Watkins,  H.,  and 
Waag ,  W . L .  Definition  of  Requirements  for  a  Performance  Measurement 
System  for  C-5  Aircrew  Members.  AFHRL-TR- 78- 54 .  Air  Force  Human 
Resources  Laboratory,  Williams  Air  Force  Base,  AZ.  October  1978. 
Waag,  Wayne  L.  and  Hubbard,  David  C.  The  Measurement  of  C-5A  Performance. 
In  Proceedings  of  Psychology  DOD  Symposium,  U.S.  Air  Force  Academy, 
April  1984. 


F-14  ISS 

F-14  Instructional  Support  System  (ISS)  Final  Technical  Report  (6/30/82) 
F-14  ISS  Operational  Design  (not  dated) 

F-14  ISS  System  Development  Notebook  Vol.  I  (not  dated) 

Program  Source  Listings 

Semple,  C.A.,  Vreuls ,  D.  ,  Cotton,  J.C.,  Durfee,  D.R.,  Hooks,  J.T.,  and 


Butler,  E.A.  The  Functional  Desien  of  an  Automated  Instructional  Support 
System  for  Operational  Flight  Trainers.  NAVTRAEQUIPCEN  76-C-0096-1. 
Naval  Training  Equipment  Center,  Orlando,  FL.  January  1979. 

Osborne,  S.R.,  Semple,  C.A. ,  and  Obermayer,  R.W.  Three  Reviews  of  the 
Instructional  Support  System  ( I S  S  )  Concept.  NAVTRAEQUIPCEN 
81 -C- 0081-1.  Naval  Training  Equipment  Center,  Orlando,  FL.  March 
1983. 

Bosworth,  L.K. ,  Kryway,  J.T.,  and  Seidensticker ,  S.S.  F-14  Instructional 
Support  System  (ISSI  Weapons  System  Training  (WST) .  NAVTRAEQUIPCEN 
80-C-0056-1.  Naval  Training  Equipment  Center,  Orlando,  FL.  March 


1981. 
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APPENDIX  F 


INTERNAL  ISS  COMPARISON 


This  appendix  presents  a_  detailed  comparison  of  internal  ISS  features 
among  the  four  systems  (AFTS®  ,  C-5A  PMS ,  F-14  ISS  and  ARPTT  ISS).  These 
systems  were  reviewed  and  examined  under  each  of  the  following  topics: 

1.  Task  module  definition 

2.  Actuation  criteria 

3.  Performance  measurement 

4.  Scoring  schemes 

5.  Instructional  support  actions 

6.  Instructor/ISS  interaction 

This  discussion  does  not  compare  the  instructor  support  feature  imple¬ 
mentation  of  these  systems;  rather,  it  describes  how  training  objectives 
were  utilized  as  controllers  and  generators  of  information. 

Task  Module  Definition 

Although  the  concept  of  a  task  module  is  used  formally  with  only  some 
of  these  systems  under  investigation,  it  can  be  generalized  to  all  of  the 
systems  by  grouping  together  system  functions  which  collectively  meet  a 
training  objective.  This  has  been  done  for  each  of  the  selected  systems  and 
the  resulting  sets  of  task  modules  are  provided  in  the  task  commonality  listing. 

Format 


The  four  systems  differ  in  the  manner  of  implementation:  The  AFTS® 

"task  modules"  are  embedded  in  Fortran  code;  F-14  ISS  and  ARPTT  ISS  are 
data-driven  by  formal  task  modules;  and,  the  C-5A  PMS  has  some  functions 
defined  in  a  fill - in-a- form  manner  but  many  other  functions  are  expressed  as 
an  elegant  general-purpose  authoring  language.  In  each  case,  however,  there 
is  some  means  to  control  the  decisions  and  actions  of  the  ISS. 

The  general  form  of  the  F-14  ISS  and  ARPTT  ISS  task  modules  is  that  of 
(a)  a  header  that  includes  identifying  information,  start/stop  logic,  and 
scoring  factors  applying  to  the  entire  task  module,  followed  by  (b)  logic, 
algorithms  and  diagnostic  feedback  appropriate  to  each  step  or  event  in  the 
task  module.  These  shall  be  further  amplified  in  a  subsequent  paragraph.  The 
C-5A  PMS  treats  specification  of  navigational  profiles  (i.e.,  instrument 
departure,  enroute ,  holding  pattern,  initial  approach,  and  ILS)  with  a  form  » 
that  resembles  the  task  module  form,  but  treats  the  remainder  of  specification 
with  a  block-structured  language  with  many  control  and  action  options;  the 
overall  result  resembles  multiple  nested  subroutines  which  give  the  author 
detailed  control  over  the  ISS. 


Flight  Tasks 

The  flight  task  modules  for  the  F-14  ISS  use  the  following  format  for 
header  information: 


1.  Type:  normal,  emergency,  tactical  (approach,  departure,..) 

2.  Name:  operational  name  for  the  specific  module  (HI  TACAN  .  ..) 

3.  Description:  a  concise  summary 

4.  Start  Conditions:  the  conditions  which  start  the  module 

5.  Stop  Conditions:  the  conditions  which  stop  the  module 

6.  Performance  Measurement/Scoring :  a  listing  of  the  steps  in  the 

task  module  and  weighting/scoring  factors. 

Each  task  module  is  further  broken  down  into  steps  (measurable  events) 
of  the  following  format: 

1.  Step  no . :  a  unique  sequence  number 

2.  Description:  statement  of  performance  requirements 

3.  Start  Conditions:  the  conditions  which  start  measurement  of  the 

step . 

4.  Stop  Conditions:  the  conditions  which  stop  the  step 

5.  Diagnostics:  immediate  feedback  of  performance,  or  other  instruc¬ 
tional  actions,  based  on  measurement  within  the  step. 

Consequently,  the  t.otal  task  module  is  composed  of  the  header  and  any 
number  of  steps  or  events,  and  in  this  manner,  quite  general  flight  task  modules 
can  be  specified. 

For  comparison,  an  After  Takeoff  Climb  Task  Module  for  the  C-5A  PMS 
consists  of  the  following  statements: 

AFTER  TAKE  OFF  CLIMB 

MONITOR  "AFTER  TAKEOFF/CLIMB"  CHECKLIST 
MONITOR  "ABQ- NORTON  ENROUTE"  ENROUTE  PROFILE 
WHEN  ALTITUDE  >  10000 

OR  MANUAL- CABIN -PRESS  <  -10 
OR  MANUAL- CABIN -PRESS  >  10  THEN 
AFTER  120  SECONDS 
ENTER  MALFUNCTION  950 

MALFUNCTION  TEXT:  "WHEN  ALTITUDE  EXCEEDS  10000  FT 
MALFUNCTION  TEXT:  "OR  MANUAL  CABIN  PRESS  IS  NOT  NORM" 
MALFUNCTION  TEXT:  "THEN  AFTER  2  MINUTES" 

MALFUNCTION  TEXT:  "  ENTER  MALFUNCTION  950" 

WHEN  M950  -  "ACTIVE"  THEN 
AFTER  60  SECONDS 
CLEAR  MALFUNCTION  950 

MALFUNCTION  TEXT:  "WHEN  MALFUNCTION  950  IS  SET  IN" 
MALFUNCTION  TEXT:  "THEN  AFTER  1  MINUTE  " 

MALFUNCTION  TEXT:  "  CLEAR  MALFUNCTION  950" 

The  enroute  profile  referenced  in  the  above  would  be  defined  through  the 
use  of  a  form  shown  in  Figure  F-l;  the  referenced  checklist  will  be  described 
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Figure  F-l.  Enroute  Route  Structure  Definition  Form  Sheets  (Sheet  5  of  5) 


Procedural  Tasks 


The  F-14  ISS  procedural  task  modules  are  of  the  following  form  for 
header  information: 

1.  Type:  Procedures  task  module  (normal,  emergency,  weapons) 

2.  Name:  Specific  name  (e.g,  takeoff  checklist) 

3.  Start  conditions:  conditions  for  starting  the  task  module 

4.  Stop  conditions:  conditions  for  stopping  the  task  module 

5.  Scoring:  Measurement  and  scoring  is  done  at  the  task  module  level 
rather  that  at  the  step  level.  Measurement  consists  of  critical 
event  measures  (e.g.,  errors  of  omission  or  commission),  mandatory 
measures  (specific  important  switches  or  events),  optional  measures, 
and  sequence  measures. 

The  remaining  parts  of  the  task  module  describe  the  steps  as  follows: 

1.  Step  no.:  unique  sequence  number 

2.  Description:  statement  of  the  procedural  activity 

3.  Contingencies:  the  events  which  must  have  taken  place  prior  to  the 
initiation  of  this  step. 

4.  Events:  a  list  of  events  appropriate  to  this  step,  including  steps 
which  are  "correct"  and  those  which  are  "incorrect" 

5.  Diagnostics:  feedback  to  the  instructor  indicating  incorrect 

actions 

In  comparison,  the  procedural  modules  definition  form  for  the  C-5A  PMS 
is  shown  in  Figure  F-2,  and  a  sample  procedural  specification  for  the  Cruise 
Checklist  is  shown  in  Figure  F-3.  The  two  methods  of  task  module  specifi¬ 
cation  are  quite  similar,  although  the  C-5  PMS  method  provides  a  method  for 
crew/individual  measurement,  and  a  different  scheme  for  specifying  sequences  of 
actions.  In  that  scheme,  steps  are  grouped  into  blocks  identified  by  BEGIN 
and  END  labels.  Blocks  may  be  nested  within  other  blocks,  and  each  block  is 
delimited  BEGIN  SEQ  .  .  .  END  or  BEGIN  NSQ  .  .  .  END  to  indicate  whether  the 
included  steps  are  to  be  performed  sequentially  or  in  arbitrary  order.  When 
this  scheme  is  used,  one  does  not  have  to  explicitly  identify  specific  actions 
which  must  precede  a  specific  event. 

Concurrent  Modules 


In  practice,  task  modules  may  occur  in  sequence,  one  after  the  other, 
as  TAKEOFF  follows  TAXI,  etc.  However,  task  modules,  or  steps  within  them, 
may  occur  concurrently,  in  parallel  or  overlapping  fashion.  In  particular,  it 
may  be  desirable  to  define  an  "umbrella"  task  module  that  is  active  during  an 
entire  training  mission;  this  module  would  continuously  test  for  abnormal 
conditions  which  might  occur  at  any  time  (e.g,  crash,  over-g  of  aircraft, 
navigation  outside  the  defined  gaming  area) . 
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;ure  F-2.  Procedural  Modules  Definition  Form  (Sheet  3  of  5). 
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Figure  F-3.  Sample  Procedure  Specification 


Other  Modules 


Flight  task  modules  and  Procedural  task  modules  are  two  examples  of  the 
task  module  concept,  and  depending  on  their  definition,  can  be  sufficient  to 
define  modules  for  training.  However,  the  implementer  may  wish  to  create 
other  categories  of  task  modules  to  suit  the  purposes  of  a  specific  design. 
For  example,  in  the  C-5A  PMS  implementation,  a  distinction  is  made  between 
checklists,  procedures,  navigational  profiles,  and  aircraft  parameter 
envelopes.  Additionally,  a  format  is  specified  for  malfunction  insertion  as 
shown  in  Figure  F-4. 

Many  of  these  distinctions  appear  to  be  implementation-specific  and 
should  be  made  at  the  discretion  of  the  designer;  however,  at  the  time  of 
specification,  it  may  be  wise  to  maintain  independence  from  design  details.  It 
may  be  desirable,  therefore,  to  use  a  simpler  task  module  format  at  the  time  of 
initial  specification.  The  following  format  was  used  in  this  analysis  in  the 
attempt  to  derive  a  common  basis  for  comparison  between  the  four  selected 
systems : 

1.  Step:  a  unique  sequence  number  (zero  for  the  header,  1..  for 

individual  steps  in  the  task  module 

2.  Description:  text  describing  the  task  or  step  to  be  performed 

3.  Start  logic:  a  concise  description  of  the  logic  which  can  be  used 
to  identify  the  starting  point  for  the  task  or  step 

4.  Stop  logic:  a  concise  description  of  the  logic  which  can  be  used  to 
identify  the  stopping  point 

5.  Measurement:  specific  measures  to  be  computed 

6.  Scoring:  a  method  for  scoring/grading  performance 

7.  Action:  actions  to  be  taken  during  the  task/step  (e.g.,  diagnostic 
'-.essages  to  be  sent  to  the  instructor,  malfunctions  to  be  inserted 

8.  Comment:  any  comments  on  the  above 

For  purposes  of  initial  specification,  the  above  format  can  be  filled- in 
with  liberal  use  of  text  which  can  allow  a  single  form  to  suffice;  whereas, 
later  in  design,  a  number  of  specific  forms  may  be  required.  In  fact  the 
same  form  can  even  be  used  for  procedural  task  modules  by  using  the  BEGIN 
SEQ  -  BEGIN  NSQ  convention  of  nested  blocks. 

Actuation  Criteria 

Since  the  basic  function  of  the  kernel  ISS  is  to  monitor  ani  take  action, 
the  ability  to  respond  based  on  criteria  specified  by  the  task  module  definit¬ 
ion  is  of  central  importance.  The  actuation  criteria  for  all  four  systems 
appears  to  be  similar. 


MALFUNCTION  SPECIFICATION  WHEN  <CHECK>  THEN 

[AFTER  <number>  SECONDS] 

i <SIMPLE  MALFUNCTION> ) 

\ <MALFUNCTION  BLOCK?  f 

[<MALFUNCTION  TEXT?] 

CHECK  Arbitrary  Boolean  expression  of  up  to  four  <S TATE>a 
MALFUNCTION  BLOCK  BEGIN  <SIMPLE  MALFUNCTION 

WHEN  <CHECK>  THEN  [AFTER  <number>  SECONDS]]  <SIMPLE  MALFUNCTION? 


The  basic  structure  for  actuation  criteria  is  of  the  form: 


<simulator  variable>  Crelational  operator>  <particular  value> 

where  relational  operators  include  equal,  not  equal,  greater  than,  greater 
than  or  equal  to,  less  than,  less  than  or  equal  to,  within  range  X,Y,  out¬ 
side  range  X,Y.  More  complex  actuation  criteria  can  be  formed  by  combining 
such  relationships  into  more  complex  logical  expressions  using  AND,  OR,  NOT. 
Ordinarily  any  simulator  variable  can  be  included  in  such  relations,  including 
communications  when  the  simulation  includes  speech  generation  equipment. 

These  actuation  criteria  can  be  thought  of  as  the  nucleus  of  an  authoring 
language  for  the  generation  of  task  modules.  Additional  refinements  to  the 
basic  actuation  criteria  structure  to  allow  implementation  of  a  wide  range 
of  task  modules  include: 

1.  Latch  time:  Latch  time  is  the  length  of  time  a  state  must  exist  for 
it  to  be  recognized.  It  is  particularly  useful  for  testing  switch  actions,  to 
ignore  intermediate  positions  of  a  rotary  switch  when  moving  from  one  position 
to  another. 

2.  Monitor  time:  Monitor  time  is  the  maximum  allowable  time  for  a  task 
module  step.  After  the  specified  time, the  action  is  taken  to  be  incomplete. 

3.  WHEN  and  IF:  A  WHEN  criterion  causes  subsequent  criteria  to  be 
evaluated  continuously  as  long  as  the  criteria  are  true;  the  IF  criterion  is 
evaluated  only  once . 

4.  Closest  point  of  approach:  Particularly  useful  for  navigational 
task  modules,  passage  of  a  navigational  aid  (e.g.  TACAN)  is  taken  to  occur 
when  the  distance  to  the  NAVAID  is  at  a  minimum. 

The  author  of  a  task  module  must  specify  actuation  criteria  in  sufficient 
detail  that  ISS  actions  are  triggered  whenever  a  specific  instructional  situat¬ 
ion  occurs  and  no  other.  It  is  quite  easy,  for  example,  for  a  deviation 
and  correction  during  straight-and-level  flight  to  be  taken  as  the  turning 
segment  which  is  to  follow  it.  It  is  also  difficult  to  anticipate  all  of 
the  odd  actions  a  student  might  take.  Consequently,  the  author  must  define 
each  task  module  with  care,  and  the  specifier  of  the  ISS  must  provide  for  a 
sufficiently  rich  set  of  actuation  criteria.  Otherwise,  unfortunate  actions 
such  as  premature  triggering  of  a  task  module,  instructor  display,  or  mal¬ 
function  insertion  may  occur. 

Performance  Measurement 

One  of  the  instructor  support  features  that  can  be  triggered  based  on 
specific  actuation  criteria  is  automated  performance  measurement.  Performance 
measurement  can  be  useful  to  the  instructors  to  supplement  their  observations 
when  they  are  busy  or  unable  to  observe.  It  may  be  the  Instant -by -instant 
graphic  display  of  a  flight  profile  plotted  against  a  nominal  profile  which  is 
otherwise  unavailable.  It  can  also  be  useful  to  provide  feedback  to  the 
student  during  unsupervised  practice  or  trial  checkrides.  Performance  measure- 


ment  can  be  used  by  the  Training  Manager  to  assess  the  instructional  process. 
It  can  provide  normative  data  to  permit  comparison  of  a  student's  performance 
with  other  previous  students  at  the  same  stage  of  training.  Performance 
measurement  can  also  be  used  for  the  purposes  of  training  research.  Further, 
it  can  be  used  within  the  ISS/ATD  for  instructional  control.  Consequently, 
there  are  a  number  of  reasons  for  including  performance  measurement  in  an  ISS. 

Flight  Task  Measurement 

The  measurement  of  flight  tasks,  as  it  is  included  in  the  four  represen¬ 
tative  systems,  depends  on  whether  performance  is  to  be  measured  over  a  period 
of  time,  or  at  a  specific  instant  of  time.  If  performance  is  to  be  measured 
over  a  period  of  time,  the  following  should  be  considered: 

1.  Average  value  (an  ordinary  average,  may  use  an  absolute  value  of  a 
parameter  to  treat  +/-  variations  the  same) 

2.  Root -mean- squared  value  (based  on  a  squared  value  so  +/-  values  are 
treated  the  same,  an  indication  of  variability) 

3.  Tolerance  bands  (within  or  outside  of  a  specified  range) 

If  performance  is  to  be  measured  at  an  instant  of  time,  then  a  "snapshot" 
is  taken  of  the  value  of  flight  task  parameters  at  the  designated  time. 

Performance  criteria,  to  be  presented  along  with  student  performance, 
may  be  based  on  the  performance  of  previous  students.  This  requires  that  data 
are  accumulated  with  each  student,  statistical  analyses  are  performed,  and 
normative  performance  criteria  are  updated  at  intervals. 

For  example,  the  C-5A  PMS  makes  the  following  measurements  during  an 
instrument  departure: 

1.  Correct  NAVAID  selected?  (correctness  of  frequency  selection) 

2.  Correct  HSI  course  selected?  (correctness  of  course  on  HSI  with 
the  correct  NAVAID  selected) 

3.  Centered  CDI?  (RMS  course  deviation  in  dots) 

4.  Specified  ground  track?  (RMS  ground  track  error  in  NM) 

5.  Specified  DME  arc  track?  (Average  arc  deviation  error  in  NM) 

6.  Within  altitude  restriction?  (Altitude  error  at  checkpoint) 

Procedural  Task  Measurement 

Procedural  measurement  for  the  representative  systems  incorporates  the 
following: 

1.  Errors  of  omission  --  events  lot  performed 

2.  Errors  of  commission  --  unwanted  events  were  performed 

3.  Constraint  errors  --  at  specified  events,  flight  parameters  were 
out  of  tolerance  (e.g.,  airspeed  at  flaps  down) 

4.  Sequence  errors  --  events  out  of  sequence 

5.  Percent  of  mandatory  actions  completed 
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6. 

7. 


Percent  of  optional  actions  completed 

Time  to  first  event  --  e.g,  beginning  of  checklist 


Note  that  procedural  performance  measurement  occurs  largely  at  the  task 
module  level,  and  that  little  measurement  is  possible  (except  for  errors  of 
commission  or  constraint  errors)  at  the  point  of  each  step  in  the  procedure. 
Note  also  that  flight  task  modules  and  procedural  task  modules,  and  the 
corresponding  measurement,  are  not  necessarily  mutually  exclusive;  they  can  be 
intermixed,  and  flight  tasks  can  constitute  a  step  in  a  procedure.  Further, 
an  option  to  calculation  of  quantitative  measures  is  the  display  of  appropriate 
information  for  diagnosis  and  assessment  by  the  instructor. 

Two  notational  schemes  have  been  used  to  denote  the  order  in  which  events 
should  occur  in  a  procedure : 

1.  Grouping  items  into  nested  blocks  to  indicate  whether  the  events 
within  a  block  must  occur  in  the  designated  order  or  whether  order  is  un¬ 
important 

2.  Listing  items  which  should  occur  before  and  after  each  item  in  a 
procedure 

It  is  also  possible  to  specify  constraints  for  each  item  in  a  procedure, 
i.e.  ,  conditions  which  must  be  met  at  the  time  of  the  event  (e.g.  ,  airspeed  when 
flaps  are  lowered) . 


Scoring 

Since  a  large  number  of  performance  measures  may  be  generated  for  various 
types  of  tasks,  a  number  of  crewmembers,  many  task  modules,  and  multiple 
simulator  sessions,  a  means  for  summarizing  and  scoring  performance  may  be 
desired.  No  standard  method  for  doing  this  is  known;  however,  the  methods 
used  for  some  of  the  representative  ISSs  are  presented  in  the  following  para¬ 
graphs  . 

AFTS® 


The  training  provided  by  the  AFTS®  consisted  of  a  series  of  user- 
defined  and  preprogrammed  training  problems  with  increasing  levels  of  task 
difficulty.  Progression  in  training  proceeded  from  the  simple  to  the  com¬ 
plex.  Difficulty  was  a  combination  of  inherent  operational  complexity  and 
variables  such  as  wind  and  equipment  malfunctions.  Each  task  module  included 
a  point  structure  for  determining  the  points  to  be  awarded  for  a  specified 
band  of  performance.  The  points  awarded  were  then  differentially  weighted  in 
accordance  with  assigned  weights  to  produce  a  proficiency  score  at  the  end  of 
an  exercise.  Based  on  the  derived  score,  the  AFTS®  would  then  adapt  the 
training  so  that  the  next  exercise  was  at  an  appropriate  level  of  difficulty 
and,  thereby,  provide  individualized,  self-paced  aircrew  training. 


F- 14  ISS 


For  each  flight  task  or  procedural  task  measurement,  a  score  was  assigned 
(in  the  range  2.5  to  4.0),  then  each  score  was  multiplied  by  a  weighting 
factor,  and  all  weighted  scores  were  summed  to  produce  a  score  for  each 
task  module.  A  matrix  was  produced  for  each  task  module,  for  viewing  by  the 
instructor,  the  rows  of  which  corresponded  to  each  performance  measure  in  the 
task  module,  and  the  columns  included  the  following  information: 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 
9. 


Nominal  Value 

4.0  Range  (upper  and  lower  measurement  limits 

3.5  Range  (upper  and  lower  measurement  limits 

3.0  Range  (upper  and  lower  measurement  limits 

2.5  Range  (upper  and  lower  measurement  limits 

Measured  Value 

Number  of  Measures 
Grade 

Weight  Factor 


for  this  score) 
for  this  score) 
for  this  score) 
for  this  score) 


Using  this  output,  the  instructor  could  see  the  desired  value  of  a 
specific  performance  measure  (nominal  value) ,  the  range  of  performance  that 
would  yield  a  specific  grade,  the  measured  or  actual  value  of  performance,  the 
grade  which  this  algorithm  produced,  and  the  weighting  factor  for  combining 
each  measure  grade  into  a  grade  for  the  entire  task  module .  An  advantage 
of  this  approach  is  that  the  grading  algorithm  is  clearly  displayed,  and 
the  instructors  are  permitted  to  change  Range  definitions  or  Weight  Factors 
to  agree  with  their  subjective  standards. 


C-5A  PMS 


The  task  module  definitions  for  the  C-5A  PMS  include  the  assignment 
of  points  to  each  step  in  a  procedure,  each  parameter  envelope,  and  each 
navigational  profile;  further,  provision  is  made  to  assign  the  resulting 
performance  measures  to  one  or  more  crewmembers.  For  example,  a  procedure  may 
yield  all  "possible  points"  if  performed  correctly,  half  of  the  "possible 
points"  if  there  is  a  sequence  violation,  and  no  points  if  an  omission  or 
constraint  violation  occurs.  Further,  the  earned  points  are  multiplied  by  a 
criticality  factor  for  each  step  to  reflect  differences  in  the  seriousness  of 
errors . 


Based  on  the  points  produced  by  the  previous  method,  five  levels  of 
proficiency  assessment  are  derived: 

1.  Performance  Monitorable  Task  Assessment  --  Combination  of  individual 
scores  into  a  single  score  for  the  performance  monitorable  task.  There  will 
be  a  separate  score  for  each  crew  member/crew  coordination  associated  with  the 
task. 


2.  Performance  Monitorable  Task  Group  Assessment  --  Combination  of 
scores  for  all  tasks  belonging  to  the  same  group  (procedure,  navigational 
profile,  parameter).  There  will  be  a  separate  score  for  each  session. 
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3.  Crew  Member/Crew  Coordination  Assessment  --  Combination  of  the 
summary  performance  scores  from  Level  2  for  a  crew  member  or  for  crew  coordi¬ 
nation. 

4.  Session  Assessment  --  Combination  of  the  pilot,  copilot,  flight 
engineer,  and  crew  coordination  proficiency  assessment  score  into  a  single 
score  for  the  session. 

5.  Mission  Assessment  --  Combination  of  the  proficiency  assessment 
scores  for  both  sessions. 

Note  that  each  mission  of  C-5A  training  is  divided  into  two  similar 
sessions;  ordinarily,  one  student  flies  as  pilot  and  the  other  copilot  on  one 
session,  and  then  they  reverse  roles  on  the  second  session. 

ARPTT  ISS 

Minimum  proficiency  levels  were  defined  for  each  of  the  ARPTT  training 
objectives,  and  stored  as  part  of  the  task  module  definitions.  The  ISS  was 
then  capable  of  assigning  a  grade  of  proficient/non-proficient.  These  grades 
could  then  be  reviewed  with  the  instructor  during  an  evaluation  of  objectives. 
The  instructor  then  had  the  capability  to  change  the  grade  if  desired.  The 
completion  of  these  objectives  will  be  kept  in  the  student's  training  summary 
and  used  as  a  reference  of  the  student's  progress.  Overall  class  statistics 
were  also  maintained  for  review  by  curriculum  managers  of  standardization 
requirements  of  the  syllabus . 

Instructional  Support  Actions 

The  ISS  has  a  basic  capability  for  recognizing  conditions  and  triggering 
actions.  Among  the  actions  which  can  be  triggered  by  the  ISS  are  malfunction 
insertion/removal,  initiation  of  displays,  set-up  of  simulator  initial  con¬ 
ditions,  communications,  and  recording  of  data.  Each  of  the  representative 
ISSs  has  this  capability  which  differs  in  accordance  with  the  specific  appli¬ 
cation  and  not  in  any  substantial  way.  Some  of  these  features  are  briefly 
summarized  below  without  attempt  to  contrast  the  four  systems. 

Malfunction  Control 

A  number  of  malfunctions  are  ordinarily  possible  in  a  modern  flight 
simulator,  and  the  ISS  may  be  required  to  control  any  of  them.  These  include 
malfunctions  that  are  controlled  (in  the  C-5A  simulator)  by  digital  entry, 
control  pots,  pot  selectors,  switches,  circuit  breakers,  and  environmental 
inputs.  Malfunction  insertion  and  removal  may  be  based  on  specific  combinations 
of  flight,  environmental  conditions,  and  time.  During  the  time  that  a  malfunc¬ 
tion  is  active,  it  may  also  be  desired  to  vary  (gradually  increase/decrease, 
or  fluctuate)  the  level  of  specific  parameters.  Control  of  malfunctions  by 
the  ISS  offers  the  potential  of  reducing  the  complexity  of  malfunction  control 
by  the  instructor  and  the  potential  for  freeing  the  instructor  for  other 
instructional  duties. 
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Initiation  of  Displays 


Based  on  actuation  criteria  stored  as  part  of  the  task  module  definition, 
the  ISS  can  trigger  the  generation  of  displays  without  direct  action  by  any 
personnel.  This  also  offers  the  potential  for  unburdening  instructional 
personnel  of  operator  duties  and  freedom  for  instruction.  Among  the  types  of 
displays  that  can  be  produced  are: 

1.  Mission  sequence  displays  (sequences  of  tasks) 

2.  Mission  plot  displays  (ground-referenced  graphics) 

3.  Route  charts  (displays  for  departure,  enroute,  approach  plates) 

4.  Checklist/procedure  displays  (displays  of  predefined  sequences 
together  with  time -stamped  crew  actions) 

5.  Error  alert  displays  (messages  alerting  instructor  to  errors) 

6.  Proficiency  assessment  displays  (scores/grades  for  specific  tasks 
and  total  mission) 

7.  Debriefing  report  (data  to  support  the  debriefing  period) 

Record  Data 

The  ISS  may  be  viewed  as  an  information  generator,  of  which  some 
information  is  generated  to  support  training- in-progress,  and  other  information 
is  generated  to  support  external  training  processes.  Among  the  external  needs 
that  may  be  supported  are  debriefing,  development  of  performance  norms, 
training  management  data  analysis,  and  training  research  data  analysis.  An 
example  of  statistics  accumulated  by  the  ARPTT  is  shown  in  Table  F-l.  Data 
for  these  needs  will  depend  on  the  specific  overall  training  system,  but  will 
certainly  include  all  recording  of  all  events  in  a  manner  appropriate  for 
debriefing,  and  will  include  all  basic  data  used  in  developing  grades  for 
proficiency  assessment.  Specific  research  requirements  may  dictate  that  even 
greater  performance  measurement  detail  be  recorded. 


Table  F-l.  Precontact  Statistics 


!  STUDENT  TYPE  ! 

! 

REQUIRE 

! 

IQP  ! 

PUP 

!  REQUAL 

! 

TOTAL 

NUMBER  OF  STUDENTS 

! 

XXXXXXX 

! 

3  ! 

0 

0 

! 

3 

AVG.  TIME  TO  PROF. 

! 

XXXXXXX 

! 

198.7  1 

0.0 

0.0 

1 

198.7 

%  TIME  OUT  IN  ALT. 

! 

0 

! 

0.0  1 

0.0 

0.0 

! 

0.0 

«  TIME  OUT  IN  AZI . 

! 

0 

! 

0.0  ! 

0.0 

0.0 

! 

0.0 

%  TIME  OUT  IN  DIST. 

! 

0 

! 

0.0  ! 

0.0 

0.0 

! 

0.0 
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Much  of  the  foregoing  discussion  is  based  on  preprogrammed  automatic 
operation  of  the  ISS.  Although  this  mode  has  a  number  of  advantages,  it  could 
provide  an  inflexible  environment  for  instruction  unless  options  for  instructor 
interaction  and  override  are  provided.  Furthermore,  not  all  crew  actions 
(e.g.t  communications,  out-of-cockpit  visual  objects)  can  be  automatically 
sensed;  in  such  cases,  the  instructor  must  provide  the  needed  information. 
Each  of  the  four  systems  provides  some  means  for  deviation  from  fully  pre¬ 
programmed  and  automatic  operation. 

ATCS® 

The  automated- adaptive  mode  is  the  normal  mode  for  AFTS®f  however,  it 
does  allow  the  instructor  to  control  the  sequence  of  training.  Adaptive 
training  can  be  modified  by  beginning  a  training  session  with  a  selection  from 
a  menu  of  initial  conditions.  For  a  given  set  of  initial  conditions,  AFTS®can 
be  paused  with  a  FREEZE  command,  and  either  CONTINUE  or  RESET  to  the  start  for 
rerun. 


Both  the  ARPTT  and  F14  ISS  provide  options  for  selecting  a  fully 
preprogrammed  mode  (CANNED  mode),  part- task  training  (PTT)  mode,  or  instructor 
constructed  sessions  produced  from  a  menu  of  task  modules  (ISSM  or  ISEL 
modes)  .  Preprogrammed  insertion  of  malfunctions  can  be  modified  through 
use  of  ACTIVATE/  REMOVE  MALFUNCTION  switches .  Control  of  task  module  sequencing 
can  be  modified  by  selection  of  reposition  options  to  SLEW  TO  or  RADAR  VECTOR 
TO,  allowing  training  to  begin  from  a  new  position. 

SLSAugMS 

The  C-5A  PMS  provides  the  instructor  with  the  capability  to  intervene 
in  the  pre-defined  sequence  of  events  and  allows  modification  of  the  selection 
of  checklists,  procedures  and  malfunctions,  and  alteration  of  the  selection  of 
displays  of  mission  information.  During  training,  the  instructor  may  exercise 
controls  to  STOP  PMS,  START,  SUSPEND,  CONTINUE,  and  SCORE.  ENTER,  CLEAR, 
START,  and  TERMINATE  controls  are  provided  to  control  the  insertion  and  removal 
of  malfunctions. 


The  preceding  discussion  is  only  exemplary  and  does  not  give  a  full 
presentation  of  the  options  for  control  provided  for  each  system.  It  is 
however  intended  to  be  suggestive  of  types  of  control  which  the  instructor  may 
need  and  to  be  suggestive  of  the  interaction  that  must  be  specified  for  each  new 
ISS.  The  ISS  must  unburden  the  instructor,  and  may  do  this  through  the  use  of 
automation;  nevertheless,  the  instructor  must  be  able  to  conveniently  override 
and  re-direct  the  system. 


A  view  has  been  taken  that  the  ISS  is  a  programmable  controller  and  a 
generator  of  information,  and  although  the  four  representative  ISSs  vary  in 
the  way  they  are  implemented,  all  fit  the  same  general  model.  The  method  of 
specifying  the  program  for  the  ISSs  varies,  but  the  task  module  concept  can  be 
used  for  initial  specification  for  any  ISS.  This  implies  a  data-driven 
system,  but  specification  in  this  manner  still  permits  a  large  degree  of 
design  freedom.  The  types  of  actuation  criteria  included  in  an  ISS  determines 
how  "smart"  the  ISS  can  be  in  behaving  "intelligently"  in  controlling  instruc¬ 
tional  events  and  features .  The  ISS  can  be  the  generator  of  a  large  amount  of 
different  types  of  information,  and  this  capability  depends  on  the  manner  in 
which  performance  measurement  is  implemented  and  the  types  of  data  recorded 
and  displayed.  Although  all  four  of  the  systems  provide  a  preprogrammed 
automatic  mode,  each  provides  some  manner  for  instructor  control  and  override, 
providing  a  degree  of  flexibility  in  use  of  the  ISS  for  tailored  instruction. 
All  provide  instructional  support  through  the  control  of  performance  measure¬ 
ment,  scoring,  displays,  malfunctions,  communications  and  data  recording.  These 
four  systems  provide  a  range  of  examples  that  characterize  the  current  tech¬ 
nology  and  provide  a  basis  for  the  generation  of  future  specifications. 


GLOSSARY  OF  TERMS 


AIRCREW  TRAINING  DEVICE  (ATD) :  A  term  that  refers  to  synthetic  training  devices 
(simulators)  used  in  support  of  aircrew  training  programs.  These  devices 
range  from  simple  procedures  trainers  to  more  complex  training  systems. 

ALGORITHM:  A  precise  characterization  of  a  method  for  solving  a  problem  or 

achieving  a  goal,  e.g.,  a  sequence  of  actions  terminating  in  a  solution. 

BRIEF:  Review  of  events,  objectives  and  procedures  with  aircrew  and  instruc¬ 

tional  staff  prior  to  simulator  session. 

CHECKLIST:  A  series  of  distinct  actions  to  be  performed  at  discrete  times. 

CHECKRIDE:  A  mission  or  profile  in  which  the  computer  monitors  the  student 

performance,  usually  from  takeoff  to  final  landing,  without  intervention  by  the 
instructor . 


CONTINUATION  TRAINING:  Training  conducted  routinely  in  operational  squadrons, 
or  proficiency  training  conducted  periodically. 

CONVERSION  TRAINING:  Initial  qualifying  training  for  a  particular  type 

of  weapon  system. 

DATA-DRIVEN:  A  system  that  relies  on  general  software  which  acts  upon  a 

database,  such  that  a  change  to  the  database  would  not  affect  a  change  to 
the  software. 


DEBRIEF:  Review  of  event  results  with  aircrew  and  instructors  subsequent 

to  simulator  session. 

INITIAL  CONDITIONS  (I.C.s):  A  set  of  conditions  or  starting  points  for  each 
training  scenario.  These  include  variables  such  as  airspeed,  altitude,  fuel 
load,  etc. 

INITIALIZATION:  Initialization  involves  specifying,  usually  from  the  instruc¬ 

tor/operator  console,  the  parameters  of  interest  and  their  values  for  posi¬ 
tioning  and  configuring  an  ATD  within  a  gaming  area. 

INSTRUCTOR  SUPPORT  FEATURE  (ISF):  Features  provided  by  the  Instructional 
Support  System  (IS)  to  aid  the  ATD  instructor  in  conducting  the  training 
exercise . 

INSTRUCTIONAL  SYSTEMS  DEVELOPMENT  (ISD) :  Procedural  approach  to  the  analysis 
of  training  requirements  and  the  development  of  training  programs  and  systems. 

INSTRUCTOR/OPERATOR  STATION  (IOS) :  The  aircrew  training  device  man-machine 
interface  where  active  control  and  monitoring  of  training  events  occurs. 
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INSTRUCTOR  SUPPORT  SYSTEM  (ISS):  Automated  system  within  the  ATD  designed  to 
aid  the  instructor  in  performing  the  training  function. 

MISSION  ESSENTIAL  NEEDS  STATEMENT  (MENS):  A  statement  prepared  by  HQ  USAF 
to  identify  and  support  the  need  for  a  new  or  improved  mission  capability. 
It  is  normally  based  on  one  or  more  SONs  and  is  prepared  if  the  Secretary 
of  the  Air  Force  or  Secretary  of  Defense  must  approve  the  need  and  the  solution 
approach. 

MODULARITY:  Property  of  a  system  which  allows  individual  units  to  be  added, 

deleted,  or  modified,  without  affecting  remainder  of  that  system. 

OFF-BOARD  STATION:  Instructor/operator  station  which  is  outside  cockpit. 

OFF-LINE:  Any  action  not  associated  with  active  training  on  the  simulator. 

ON-BOARD  STATION:  Instructor/operator  station  which  is  inside  cockpit. 

ON-LINE:  Controlled  directly  by  a  computer,  usually  in  ‘association  with 

active  training. 

OPERATIONAL  FLIGHT  TRAINER  (OFT):  A  device  that  dynamically  simulates  the 
flight  characteristics  of  the  designated  aircraft  to  train  flight  crews  in 
cockpit  procedures,  instrument  flight  procedures,  emergency  procedures,  commu¬ 
nications  and  navigation  procedures,  and  includes  limited  mission  execution 

PART-TASK  TRAINER  (PTT) :  A  device  which  provides  selected  aspects  of  a  task 
(fuel  system  operation,  air  refueling,  radar  operations,  etc.)  to  be  practiced 
and  a  high  degree  of  skill  developed  independently  of  other  mission  tasks. 

PERFORMANCE  MEASUREMENT  SYSTEM  (PMS) :  The  computer-based  monitoring,  recording, 
processing  and  displaying  of  objective,  quantitative  information  for  describing 
and  diagnosing  student  performance. 

PROGRAM  MANAGEMENT  DIRECTIVE  ( PMD ) :  The  official  HQ  USAF  management  directive 
used  to  provide  direction  to  the  implementing  and  participating  commands 
and  to  satisfy  documentation  requirements. 

PROGRAMMED  MISSION  SCENARIOS:  Highly  structured  sets  of  events  that  are 
caused  to  occur  automatically,  under  computer  control. 

SAMPLING  RATE:  The  temporal  frequency  at  which  a  stated  variable  (parameter) 
may  be  recorded  or  examined  by  an  automated  performance  measurement  system. 

SCENARIO:  A  predefined  sequence  of  training  events  used  to  exercise  the 

capabilities  of  an  ATD  in  a  specific  area  of  intended  training  usage. 

SPECIFICATION:  Statement  describing  the  device  to  be  built  in  terms  of  its 

functions  and  characteristics . 

STATEMENT  OF  OPERATIONAL  NEED  (SON):  A  general  statement  of  requirements 

prepared  by  one  of  the  Air  Force  Major  Commands. 
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SYLLABUS:  Course  of  study 

TASK  MODULE:  User -oriented  building  blocks  that  correspond  to  the  operational 
training  requirements  which  have  a  direct  correlation  to  a  group  of  files 
which  make  up  the  data  base  for  a  modular  data  base  driven  system. 

TRAINING  OBJECTIVES:  Explicit  statements  of  the  goals  of  training  including 
tasks  to  be  performed,  the  performance  standards  for  each  task,  and  the 
conditions  under  which  those  tasks  are  to  be  performed. 

TRAINING  REQUIREMENTS:  General  statements  of  task  performance  skills  required 
for  operational  proficiency. 

UNDERGRADUATE  PILOT  TRAINING:  Initial  pilot  flight  training. 

WEAPON  SYSTEM  TRAINER  (WST) :  A  device  which  provides  a  synthetic  flight 
and  tactics  environment  in  which  aircrews  learn,  develop,  and  improve  the 
techniques  associated  with  their  crew  position  in  a  specific  aircraft,  and 
operate  individually  or  as  a  team  in  the  execution  of  simulated  missions. 
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ABBREVIATIONS  AND  ACRONYMS 


AAI 

air-to-air  intercept 

AFB 

Air  Force  Base 

AFHRL 

Air  Force  Human  Resources  Laboratory 

AFLC 

Air  Force  Logistics  Command 

AFR 

Air  Force  Regulation 

AFSC 

Air  Force  Systems  Command 

AFTS® 

Automated  Flight  Training  System 

ARPTT 

aerial  refueling  part- task  trainer 

ASD 

Aeronautical  Systems  Division 

ATC 

Air  Training  Command 

ATD 

aircrew  training  device 

BVR 

beyond  visual  range 

CCA 

carrier  controlled  approach 

CDRL 

contract  data  requirements  list 

CPT 

cockpit  procedures  trainer 

CRT 

cathode  ray  tube 

DBMS 

data  base  management  system 

DO 

Director  of  Operations 

DR 

Director  of  Requirements 

DRF 

Dual  Role  Fighter 

ENET 

Engineering  Equipment  and  Training 

GAR 

ground  attack  radar 

GAT 

ground  attack  tactical 

GCA 

ground  controlled  approach 

HQ  USAF 

Headquarters,  United  States  Air  Force 

Hz 

hertz 

I.C. 

initial  condition 

ILS 

Instrument  Landing  System 

I/O 

input/output 

IOS 

instructor/operator  station 

IOT&E 

Initial  operational  test  and  evaluation 

IP 

instructor  pilot 

ISD 

instructional  system  development 

ISF 

instructor  support  feature 

ISS 

Instructor  Support  System 

KB 

kilobyte 
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MAC 

MAJCOM 
MB 
MCU 
MENS 
MIL- STD 


OFT 

OT&E 

PIDS 

PM 

PMD 

PMP 

PMS 

PTT 

R&D 

SAC 

SECU 

SID 

SIMCERT 

SimSPO 

SLOC 

SON 

SOW 


TAC 

TACAN 

TM 

UPT 

WST 


Military  Airlift  Command 

major  command 

megabyte 

malfunction  control  unit 
Mission  Element  Need  Statement 
Military  Standard 


operational  flight  trainer 
operational  test  &  evaluation 

prime  item  development  specification 
program  manager 
Program  Management  Directive 
Program  management  Plan 
performance  measurement  system 
part- task  trainer 

research  and  development 

Strategic  Air  Command 
simulation  exercise  control  unit 
standard  instrument  departure 
Simulator  Certification  Program 
Simulator  Systems  Program  Office 
source  lines  of  code 
Statement  of  Need 
Statement  of  Work 


Tactical  Air  Command 
tactical  air  navigation 
task  module 

Undergraduate  Pilot  Training 
weapon  system  trainer 
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